On 28.07.2014 23:32, Eelco Hoogendoorn wrote:
> I see, thanks for the clarification. Just for the sake of argument,
> since unfortunately I don't have the time to go dig in the guts of numpy
> myself: a design which always produces results of the same (high)
> accuracy, but only optimizes the commo
I see, thanks for the clarification. Just for the sake of argument, since
unfortunately I don't have the time to go dig in the guts of numpy myself:
a design which always produces results of the same (high) accuracy, but
only optimizes the common access patterns in a hacky way, and may be
inefficie
On Mo, 2014-07-28 at 16:31 +0200, Eelco Hoogendoorn wrote:
> Sebastian:
>
>
> Those are good points. Indeed iteration order may already produce
> different results, even though the semantics of numpy suggest
> identical operations. Still, I feel this different behavior without
> any semantical cl
Sebastian:
Those are good points. Indeed iteration order may already produce different
results, even though the semantics of numpy suggest identical operations.
Still, I feel this different behavior without any semantical clues is
something to be minimized.
Indeed copying might have large speed i
On Mo, 2014-07-28 at 15:50 +0200, Fabien wrote:
> On 28.07.2014 15:30, Daπid wrote:
> > An example using float16 on Numpy 1.8.1 (I haven't seen diferences with
> > float32):
>
> Why aren't there differences between float16 and float32 ?
>
float16 calculations are actually float32 calculations. I
On Mo, 2014-07-28 at 15:35 +0200, Sturla Molden wrote:
> On 28/07/14 15:21, alex wrote:
>
> > Are you sure they always give different results? Notice that
> > np.ones((N,2)).mean(0)
> > np.ones((2,N)).mean(1)
> > compute means of different axes on transposed arrays so these
> > differences 'cance
On 28.07.2014 15:30, Daπid wrote:
> An example using float16 on Numpy 1.8.1 (I haven't seen diferences with
> float32):
Why aren't there differences between float16 and float32 ?
Could this be related to my earlier post in this thread where I
mentioned summation problems occurring much earlier i
On 28/07/14 15:21, alex wrote:
> Are you sure they always give different results? Notice that
> np.ones((N,2)).mean(0)
> np.ones((2,N)).mean(1)
> compute means of different axes on transposed arrays so these
> differences 'cancel out'.
They will be if different algorithms are used. np.ones((N,2)
On 28 July 2014 14:46, Sebastian Berg wrote:
> > To rephrase my most pressing question: may np.ones((N,2)).mean(0) and
> > np.ones((2,N)).mean(1) produce different results with the
> > implementation in the current master? If so, I think that would be
> > very much regrettable; and if this is a m
On Mon, Jul 28, 2014 at 8:46 AM, Sebastian Berg
wrote:
> On Mo, 2014-07-28 at 14:37 +0200, Eelco Hoogendoorn wrote:
>> To rephrase my most pressing question: may np.ones((N,2)).mean(0) and
>> np.ones((2,N)).mean(1) produce different results with the
>> implementation in the current master? If so,
On Mo, 2014-07-28 at 14:37 +0200, Eelco Hoogendoorn wrote:
> To rephrase my most pressing question: may np.ones((N,2)).mean(0) and
> np.ones((2,N)).mean(1) produce different results with the
> implementation in the current master? If so, I think that would be
> very much regrettable; and if this is
To rephrase my most pressing question: may np.ones((N,2)).mean(0) and
np.ones((2,N)).mean(1) produce different results with the implementation in
the current master? If so, I think that would be very much regrettable; and
if this is a minority opinion, I do hope that at least this gets documented
i
Nathaniel Smith wrote:
> The problem here is that when summing up the values, the sum gets
> large enough that after rounding, x + 1 = x and the sum stops
> increasing.
Interesting. That explains why the divide-and-conquer reduction is much
more robust.
Thanks :)
Sturla
_
Thanks for the clarification, but how is the numpy rounding directed?
Round to nearest, ties to even?
http://en.wikipedia.org/wiki/IEEE_floating_point#Rounding_rules
Just curious, as I couldn't find a reference.
- Ray
At 07:44 AM 7/27/2014, you wrote:
>On Sun, Jul 27, 2014 at 3:16 PM, RayS wro
On Sun, Jul 27, 2014 at 3:16 PM, RayS wrote:
> At 02:04 AM 7/27/2014, you wrote:
>
>>You won't be able to do it by accident or omission or a lack of
>>discipline. It's not a tempting public target like, say, np.seterr().
>
> BTW, why not throw an overflow error in the large float32 sum() case?
> I
At 02:04 AM 7/27/2014, you wrote:
>You won't be able to do it by accident or omission or a lack of
>discipline. It's not a tempting public target like, say, np.seterr().
BTW, why not throw an overflow error in the large float32 sum() case?
Is it too expensive to check while accumulating?
- Ray
On Sun, Jul 27, 2014 at 9:56 AM, wrote:
>
> On Sun, Jul 27, 2014 at 4:24 AM, Robert Kern wrote:
>>
>> On Sun, Jul 27, 2014 at 7:04 AM, wrote:
>> >
>> > On Sat, Jul 26, 2014 at 5:19 PM, Sturla Molden
>> > wrote:
>> >>
>> >> Robert Kern wrote:
>> >>
>> >> >> It would presumably require a globa
On Sun, Jul 27, 2014 at 4:24 AM, Robert Kern wrote:
> On Sun, Jul 27, 2014 at 7:04 AM, wrote:
> >
> > On Sat, Jul 26, 2014 at 5:19 PM, Sturla Molden
> > wrote:
> >>
> >> Robert Kern wrote:
> >>
> >> >> It would presumably require a global threading.RLock for protecting
> the
> >> >> global st
On Sun, Jul 27, 2014 at 7:04 AM, wrote:
>
> On Sat, Jul 26, 2014 at 5:19 PM, Sturla Molden
> wrote:
>>
>> Robert Kern wrote:
>>
>> >> It would presumably require a global threading.RLock for protecting the
>> >> global state.
>> >
>> > We would use thread-local storage like we currently do with
On Sat, Jul 26, 2014 at 5:19 PM, Sturla Molden
wrote:
> Robert Kern wrote:
>
> >> It would presumably require a global threading.RLock for protecting the
> >> global state.
> >
> > We would use thread-local storage like we currently do with the
> > np.errstate() context manager. Each thread will
Robert Kern wrote:
>> It would presumably require a global threading.RLock for protecting the
>> global state.
>
> We would use thread-local storage like we currently do with the
> np.errstate() context manager. Each thread will have its own "global"
> state.
That sounds like a better plan, yes
On Sat, Jul 26, 2014 at 8:04 PM, Sturla Molden wrote:
> Benjamin Root wrote:
>
>> My other concern would be with multi-threaded code (which is where a global
>> state would be bad).
>
> It would presumably require a global threading.RLock for protecting the
> global state.
We would use thread-lo
I completely agree with Eelco. I expect numpy.mean to do something
simple and straightforward. If the naive method is not well suited for
my data, I can deal with it and have my own ad hoc method.
On Sat, Jul 26, 2014 at 3:19 PM, Eelco Hoogendoorn
wrote:
> Perhaps I in turn am missing something;
wrote:
> statsmodels still has avoided anything that smells like a global state that
> changes calculation.
If global states are stored in a stack, as in OpenGL, it is not so bad. A
context manager could push a state in __enter__ and pop the state in
__exit__. This is actually how I write OpenGL
Perhaps I in turn am missing something; but I would suppose that any
algorithm that requires multiple passes over the data is off the table?
Perhaps I am being a little old fashioned and performance oriented here,
but to make the ultra-majority of use cases suffer a factor two performance
penalty f
Benjamin Root wrote:
> My other concern would be with multi-threaded code (which is where a global
> state would be bad).
It would presumably require a global threading.RLock for protecting the
global state.
Sturla
___
NumPy-Discussion mailing list
On Sat, Jul 26, 2014 at 2:44 PM, Benjamin Root wrote:
> That is one way of doing it, and probably the cleanest way. Or else you
> have to pass in the context object everywhere anyway. But I am not so
> concerned about that (we do that for other things as well). Bigger concerns
> would be nested c
That is one way of doing it, and probably the cleanest way. Or else you
have to pass in the context object everywhere anyway. But I am not so
concerned about that (we do that for other things as well). Bigger concerns
would be nested contexts. For example, what if one of the scikit functions
use su
On Sat, Jul 26, 2014 at 9:57 AM, Benjamin Root wrote:
> I could get behind the context manager approach. It would help keep
> backwards compatibility, while providing a very easy (and clean) way of
> consistently using the same reduction operation. Adding kwargs is just a
> road to hell.
>
Would
Sturla Molden wrote:
> Sebastian Berg wrote:
>
>> Yes, it is much more complicated and incompatible with naive ufuncs if
>> you want your memory access to be optimized. And optimizing that is very
>> much worth it speed wise...
>
> Why? Couldn't we just copy the data chunk-wise to a temporary b
Sebastian Berg wrote:
> Yes, it is much more complicated and incompatible with naive ufuncs if
> you want your memory access to be optimized. And optimizing that is very
> much worth it speed wise...
Why? Couldn't we just copy the data chunk-wise to a temporary buffer of say
2**13 numbers and th
A context manager makes sense.
I very much appreciate the time constraints and the effort put in this far,
but if we can not make something work uniformly, I wonder if we should
include it in the master at all. I don't have a problem with customizing
algorithms where fp accuracy demands it; I have
On Sa, 2014-07-26 at 15:38 +0200, Eelco Hoogendoorn wrote:
> I was wondering the same thing. Are there any known tradeoffs to this
> method of reduction?
>
Yes, it is much more complicated and incompatible with naive ufuncs if
you want your memory access to be optimized. And optimizing that is ve
I could get behind the context manager approach. It would help keep
backwards compatibility, while providing a very easy (and clean) way of
consistently using the same reduction operation. Adding kwargs is just a
road to hell.
Cheers!
Ben Root
On Sat, Jul 26, 2014 at 9:53 AM, Julian Taylor <
jta
On 26.07.2014 15:38, Eelco Hoogendoorn wrote:
>
> Why is it not always used?
for 1d reduction the iterator blocks by 8192 elements even when no
buffering is required. There is a TODO in the source to fix that by
adding additional checks. Unfortunately nobody knows hat these
additional tests would
I was wondering the same thing. Are there any known tradeoffs to this
method of reduction?
On Sat, Jul 26, 2014 at 12:39 PM, Sturla Molden
wrote:
> Sebastian Berg wrote:
>
> > chose more stable algorithms for such statistical functions. The
> > pairwise summation that is in master now is very
Sebastian Berg wrote:
> chose more stable algorithms for such statistical functions. The
> pairwise summation that is in master now is very awesome, but it is not
> secure enough in the sense that a new user will have difficulty
> understanding when he can be sure it is used.
Why is it not alway
On Fr, 2014-07-25 at 21:23 +0200, Eelco Hoogendoorn wrote:
> It need not be exactly representable as such; take the mean of [1, 1
> +eps] for instance. Granted, there are at most two number in the range
> of the original dtype which are closest to the true mean; but im not
> sure that computing the
-
From: "Julian Taylor"
Sent: 26-7-2014 00:58
To: "Discussion of Numerical Python"
Subject: Re: [Numpy-discussion] numpy.mean still broken for largefloat32arrays
On 25.07.2014 23:51, Eelco Hoogendoorn wrote:
> Ray: I'm not working with Hubble data, but yeah the
At 02:36 PM 7/25/2014, you wrote:
>But it doesn't compensate for users to be aware of the problems. I
>think the docstring and the description of the dtype argument is pretty clear.
Most of the docs for the affected functions do not have a Note with
the same warning as mean()
- Ray
__
gave mediocre speedups (10%-20% on
an i5).
>
> From: RayS <mailto:r...@blue-cove.com>
> Sent: 25-7-2014 23:26
> To: Discussion of Numerical Python <mailto:numpy-discussion@scipy.org>
> Subject: Re: [Numpy-discussion] num
uot;RayS"
Sent: 25-7-2014 23:26
To: "Discussion of Numerical Python"
Subject: Re: [Numpy-discussion] numpy.mean still broken for largefloat32arrays
At 11:29 AM 7/25/2014, you wrote:
>On Fri, Jul 25, 2014 at 5:56 PM, RayS wrote:
> > The important point was that it would be
On Fri, Jul 25, 2014 at 4:25 PM, RayS wrote:
> At 11:29 AM 7/25/2014, you wrote:
> >On Fri, Jul 25, 2014 at 5:56 PM, RayS wrote:
> > > The important point was that it would be best if all of the
> > methods affected
> > > by summing 32 bit floats with 32 bit accumulators had the same Notes as
>
At 11:29 AM 7/25/2014, you wrote:
>On Fri, Jul 25, 2014 at 5:56 PM, RayS wrote:
> > The important point was that it would be best if all of the
> methods affected
> > by summing 32 bit floats with 32 bit accumulators had the same Notes as
> > numpy.mean(). We went through a lot of code yesterday,
It need not be exactly representable as such; take the mean of [1, 1+eps]
for instance. Granted, there are at most two number in the range of the
original dtype which are closest to the true mean; but im not sure that
computing them exactly is a tractable problem for arbitrary input.
Im not sure w
On Fri, Jul 25, 2014 at 5:56 PM, RayS wrote:
> The important point was that it would be best if all of the methods affected
> by summing 32 bit floats with 32 bit accumulators had the same Notes as
> numpy.mean(). We went through a lot of code yesterday, assuming that any
> numpy or Scipy.stats fu
On 7/25/2014 1:40 PM, Eelco Hoogendoorn wrote:
> At the risk of repeating myself: explicit is better than implicit
This sounds like an argument for renaming the `mean` function `naivemean`
rather than `mean`. Whatever numpy names `mean`, shouldn't it
implement an algorithm that produces the mean
: "RayS"
Sent: 25-7-2014 19:56
To: "Discussion of Numerical Python"
Subject: Re: [Numpy-discussion] numpy.mean still broken for largefloat32arrays
At 07:22 AM 7/25/2014, you wrote:
> We were talking on this in the office, as we
> realized it does affect a couple of
At 07:22 AM 7/25/2014, you wrote:
> We were talking on this in the office, as we
> realized it does affect a couple of lines dealing
> with large arrays, including complex64.
> As I expect Python modules to work uniformly
> cross platform unless documented otherwise, to me
> that includes 32 vs 6
On Fri, Jul 25, 2014 at 3:11 PM, RayS wrote:
> At 01:22 AM 7/25/2014, you wrote:
>> Actually the maximum precision I am not so
>> sure of, as I personally prefer to make an
>> informed decision about precision used, and get
>> an error on a platform that does not support
>> the specified precisio
At 01:22 AM 7/25/2014, you wrote:
> Actually the maximum precision I am not so
> sure of, as I personally prefer to make an
> informed decision about precision used, and get
> an error on a platform that does not support
> the specified precision, rather than obtain
> subtly or horribly broke
--
> From: Alan G Isaac
> Sent: 25-7-2014 00:10
>
> To: Discussion of Numerical Python
> Subject: Re: [Numpy-discussion] numpy.mean still broken for
> largefloat32arrays
>
> On 7/24/2014 4:42 PM, Eelco Hoogendoorn wrote:
> > This isn't a
nal Message-
From: "Alan G Isaac"
Sent: 25-7-2014 00:10
To: "Discussion of Numerical Python"
Subject: Re: [Numpy-discussion] numpy.mean still broken for largefloat32arrays
On 7/24/2014 4:42 PM, Eelco Hoogendoorn wrote:
> This isn't a bug report, but rather a feat
53 matches
Mail list logo
