Re: [Numpy-discussion] Wanted: new release manager for 1.5 and above
Charles R Harris wrote: > > > On Wed, Jan 13, 2010 at 11:34 PM, David Cournapeau > mailto:da...@silveregg.co.jp>> wrote: > > Charles R Harris wrote: > > > > > > > What is the setup one needs to build the installers? It might be > well to > > document that, the dependencies, and the process. > > Right. The top script is: > http://projects.scipy.org/numpy/browser/trunk/release.sh > > the bulk of the work is in : > http://projects.scipy.org/numpy/browser/trunk/pavement.py > > which describes what is needed to build installers. On mac os x, the > release script may be used as is to build every installer + the release > notes. > > > Umm, I think it needs some more explanation. There are virtual > environments, c compilers, wine, paver, etc. All/ of these might require > some installation, version numbers, and setup. This might all seem clear > to you, but a newbie coming on to build the packages probably needs more > instruction. I think it is a waste of time to document all this very precisely, because it is continuously changing. Documenting everything would boil down to rewrite the paver script in English (and most likely would be much more verbose). That's exactly why I was suggesting to have some volunteers to do 1.4.1 to do the release together, as a way to "pass the knowledge around". cheers, David ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Wanted: new release manager for 1.5 and above
On Wed, Jan 13, 2010 at 11:34 PM, David Cournapeau wrote: > Charles R Harris wrote: > > > > > > > What is the setup one needs to build the installers? It might be well to > > document that, the dependencies, and the process. > > Right. The top script is: > http://projects.scipy.org/numpy/browser/trunk/release.sh > > the bulk of the work is in : > http://projects.scipy.org/numpy/browser/trunk/pavement.py > > which describes what is needed to build installers. On mac os x, the > release script may be used as is to build every installer + the release > notes. > > Umm, I think it needs some more explanation. There are virtual environments, c compilers, wine, paver, etc. All/ of these might require some installation, version numbers, and setup. This might all seem clear to you, but a newbie coming on to build the packages probably needs more instruction. What sort of setup do you run, what hardware, etc. If code needs to be compiled for the PCC I assume the compiler needs to be to do that. What about c libraries (for numpy) and c++ libraries (for scipy)? Does one need a MAC? etc. I'm probably just ignorant, but I think a careful step by step procedure would be helpful. Chuck ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Matrix vs array in ma.minimum
On Fri, Jan 15, 2010 at 11:59 AM, Pierre GM wrote:
> On Jan 14, 2010, at 8:52 PM, David Cournapeau wrote:
>> Pierre GM wrote:
>>
>>>
>>> Er, no.
>>> np.ma.minimum(a, b) returns the lowest value of a and b element-wsie, or
>>> the the lowest element of a is b is None. The behavior is inherited from
>>> the very first implementation of maskedarray in numeric. This itself is
>>> unexpected, since np.minimum requires at least 2 input arguments.
>>>
>>> As you observed, the current function breaks down w/ np.matrix objects when
>>> only one argument is given (and when the axis is None): we call
>>> umath.minimum.reduce on the ravelled matirx, which returns the ravelled
>>> matrix. One would expect a scalar, so yes, this behavior is also unexpected.
>>>
>>> Now, which way should we go ? Keep np.ma.minimum as it is (fixing the bug
>>> so that a scalar is returned if the function is called with only 1 argument
>>> and an axis None) ? Adapt it to match np.minimum ?
>>
>> I am not a user of Masked Array, so I don't know what is the most
>> desirable behavior.
>
> I'm not a regular user of np.minimum.
Damn, I thought I coul
>
>> The problem appears when using pylab.imshow on
>> matrices, because matplotlib (and not matlab :) ) uses masked arrays
>> when normalizing the values.
>
>
> David, you mind pointing me to the relevan part of the code and/or give me an
> example ?
Here is a self-contained example reproducing the matplotlib pb:
import numpy as np
from numpy import ma
import matplotlib.cbook as cbook
class Normalize:
"""
Normalize a given value to the 0-1 range
"""
def __init__(self, vmin=None, vmax=None, clip=False):
"""
If *vmin* or *vmax* is not given, they are taken from the input's
minimum and maximum value respectively. If *clip* is *True* and
the given value falls outside the range, the returned value
will be 0 or 1, whichever is closer. Returns 0 if::
vmin==vmax
Works with scalars or arrays, including masked arrays. If
*clip* is *True*, masked values are set to 1; otherwise they
remain masked. Clipping silently defeats the purpose of setting
the over, under, and masked colors in the colormap, so it is
likely to lead to surprises; therefore the default is
*clip* = *False*.
"""
self.vmin = vmin
self.vmax = vmax
self.clip = clip
def __call__(self, value, clip=None):
if clip is None:
clip = self.clip
if cbook.iterable(value):
vtype = 'array'
val = ma.asarray(value).astype(np.float)
else:
vtype = 'scalar'
val = ma.array([value]).astype(np.float)
self.autoscale_None(val)
vmin, vmax = self.vmin, self.vmax
if vmin > vmax:
raise ValueError("minvalue must be less than or equal to maxvalue")
elif vmin==vmax:
return 0.0 * val
else:
if clip:
mask = ma.getmask(val)
val = ma.array(np.clip(val.filled(vmax), vmin, vmax),
mask=mask)
result = (val-vmin) * (1.0/(vmax-vmin))
if vtype == 'scalar':
result = result[0]
return result
def inverse(self, value):
if not self.scaled():
raise ValueError("Not invertible until scaled")
vmin, vmax = self.vmin, self.vmax
if cbook.iterable(value):
val = ma.asarray(value)
return vmin + val * (vmax - vmin)
else:
return vmin + value * (vmax - vmin)
def autoscale(self, A):
'''
Set *vmin*, *vmax* to min, max of *A*.
'''
self.vmin = ma.minimum(A)
self.vmax = ma.maximum(A)
def autoscale_None(self, A):
' autoscale only None-valued vmin or vmax'
if self.vmin is None: self.vmin = ma.minimum(A)
if self.vmax is None: self.vmax = ma.maximum(A)
def scaled(self):
'return true if vmin and vmax set'
return (self.vmin is not None and self.vmax is not None)
if __name__ == "__main__":
x = np.random.randn(10, 10)
mx = np.matrix(x)
print Normalize()(x)
print Normalize()(mx)
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] comparison operators (e.g. ==) on array with dtype object do not work
On Thu, Jan 14, 2010 at 3:49 PM, Warren Weckesser < warren.weckes...@enthought.com> wrote: > Yaroslav Halchenko wrote: > > Dear NumPy People, > > > > First I want to apologize if I misbehaved on NumPy Trac by reopening the > > closed ticket > > http://projects.scipy.org/numpy/ticket/1362 > > but I still feel strongly that there is misunderstanding > > and the bug/defect is valid. I would appreciate if someone would waste > > more of his time to persuade me that I am wrong but please first read > > till the end: > > > > The issue, as originally reported, is demonstrated with: > > > > ,--- > > | > python -c 'import numpy as N; print N.__version__; a=N.array([1, > (0,1)],dtype=object); print a==1; print a == (0,1), a[1] == (0,1)' > > | 1.5.0.dev > > | [ True False] > > | [False False] True > > `--- > > > > whenever I expected the last line to be > > > > [False True] True > > > > charris (thanks for all the efforts to enlighten me) summarized it as > > > > """the result was correct given that the tuple (0,1) was converted to an > > object array with elements 0 and 1. It is *not* converted to an array > > containing a tuple. """ > > > > and I was trying to argue that it is not the case in my example. It is > > the case in charris's example though whenever both elements are of > > the same length, or there is just a single tuple, i.e. > > > > > > The "problem" is that the tuple is converted to an array in the > statement that > does the comparison, not in the construction of the array. Numpy attempts > to convert the right hand side of the == operator into an array. It > then does > the comparison using the two arrays. > > One way to get what you want is to create your own array and then do > the comparison: > > In [1]: import numpy as np > > In [2]: a = np.array([1, (0,1)], dtype='O') > > In [3]: t = np.empty(1, dtype='O') > > In [4]: t[0] = (0,1) > > In [5]: a == t > Out[5]: array([False, True], dtype=bool) > > > In the above code, a numpy array 't' of objects with shape (1,) is created, > and the single element is assigned the value (0,1). Then the comparison > works as expected. > > More food for thought: > > In [6]: b = np.array([1, (0,1), "foo"], dtype='O') > > In [7]: b == 1 > Out[7]: array([ True, False, False], dtype=bool) > > In [8]: b == (0,1) > Out[8]: False > > Oooh, that last one is strange. Also In [6]: arange(2) == arange(3) Out[6]: False So the comparison isn't element-wise. I rather think a shape mismatch error should be raised in this case. Chuck ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Matrix vs array in ma.minimum
On Jan 14, 2010, at 8:52 PM, David Cournapeau wrote: > Pierre GM wrote: > >> >> Er, no. >> np.ma.minimum(a, b) returns the lowest value of a and b element-wsie, or the >> the lowest element of a is b is None. The behavior is inherited from the >> very first implementation of maskedarray in numeric. This itself is >> unexpected, since np.minimum requires at least 2 input arguments. >> >> As you observed, the current function breaks down w/ np.matrix objects when >> only one argument is given (and when the axis is None): we call >> umath.minimum.reduce on the ravelled matirx, which returns the ravelled >> matrix. One would expect a scalar, so yes, this behavior is also unexpected. >> >> Now, which way should we go ? Keep np.ma.minimum as it is (fixing the bug so >> that a scalar is returned if the function is called with only 1 argument and >> an axis None) ? Adapt it to match np.minimum ? > > I am not a user of Masked Array, so I don't know what is the most > desirable behavior. I'm not a regular user of np.minimum. > The problem appears when using pylab.imshow on > matrices, because matplotlib (and not matlab :) ) uses masked arrays > when normalizing the values. David, you mind pointing me to the relevan part of the code and/or give me an example ? In any case, I'd appreciate more feedback on the behavior of np.ma.minimum. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Matrix vs array in ma.minimum
Pierre GM wrote: > > Er, no. > np.ma.minimum(a, b) returns the lowest value of a and b element-wsie, or the > the lowest element of a is b is None. The behavior is inherited from the very > first implementation of maskedarray in numeric. This itself is unexpected, > since np.minimum requires at least 2 input arguments. > > As you observed, the current function breaks down w/ np.matrix objects when > only one argument is given (and when the axis is None): we call > umath.minimum.reduce on the ravelled matirx, which returns the ravelled > matrix. One would expect a scalar, so yes, this behavior is also unexpected. > > Now, which way should we go ? Keep np.ma.minimum as it is (fixing the bug so > that a scalar is returned if the function is called with only 1 argument and > an axis None) ? Adapt it to match np.minimum ? I am not a user of Masked Array, so I don't know what is the most desirable behavior. The problem appears when using pylab.imshow on matrices, because matplotlib (and not matlab :) ) uses masked arrays when normalizing the values. cheers, David ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] comparison operators (e.g. ==) on array with dtype object do not work
Hi Warren,
> The "problem" is that the tuple is converted to an array in the
> statement that does the comparison, not in the construction of the
> array. Numpy attempts
> to convert the right hand side of the == operator into an array.
> It then does the comparison using the two arrays.
Thanks for the description! It kinda makes sense now, although, in
general, I am not pleased with the API, I would take it as a documented
feature from now on ;)
> One way to get what you want is to create your own array and then do
> the comparison:
yeah... I might like to check if lhs has dtype==dtype('object') and then
convert that rhs item into object array before comparison (for now I
just did list comprehension ;))
> In [8]: b == (0,1)
> Out[8]: False
yeah -- lengths are different now ;)
> In [9]: b == "foo"
> Out[9]: array([False, False, True], dtype=bool)
yeah -- strings are getting special treatment despite being iterables
;) but that is ok I guess anyways
The main confusion seems to come from the feature of numpy in doing
smart things -- like deciding either it thinks it needs to do
element-wise comparison across lhs and rhs (if lengths match) or mapping
comparison across all items. That behavior is quite different from basic
Python iterable containers suchas tuples and lists, where it does just
global comparison:
,---
| *In [33]: [1,2] == [1,3]
| Out[33]: False
|
| *In [34]: array([1,2]) == array([1,3])
| Out[34]: array([ True, False], dtype=bool)
`---
I guess I just need to remember that and what you have described
thanks again
--
Yaroslav O. Halchenko
Postdoctoral Fellow, Department of Psychological and Brain Sciences
Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
Phone: +1 (603) 646-9834 Fax: +1 (603) 646-1419
WWW: http://www.linkedin.com/in/yarik
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] comparison operators (e.g. ==) on array with dtype object do not work
On Thu, 14 Jan 2010, josef.p...@gmail.com wrote: > It looks difficult to construct an object array with only 1 element, > since a tuple is interpreted as different array elements. yeap > It looks like some convention is necessary for interpreting a tuple in > the array construction, but it doesn't look like a problem with the > comparison operator just a consequence. Well -- there is a reason why we use tuples -- they are immutable ... as well as strings actually ;) Thus, imho, it would be a logical API if immutable datatypes are not coerced magically into mutable arrays at least whenever I am already requesting dtype='O'. Such generic treatment of immutable dtypes would address special treatment of strings but it is too much of a change and debatable anyways ;-) -- Yaroslav O. Halchenko Postdoctoral Fellow, Department of Psychological and Brain Sciences Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755 Phone: +1 (603) 646-9834 Fax: +1 (603) 646-1419 WWW: http://www.linkedin.com/in/yarik ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] comparison operators (e.g. ==) on array with dtype object do not work
On Thu, Jan 14, 2010 at 5:49 PM, Warren Weckesser
wrote:
> Yaroslav Halchenko wrote:
>> Dear NumPy People,
>>
>> First I want to apologize if I misbehaved on NumPy Trac by reopening the
>> closed ticket
>> http://projects.scipy.org/numpy/ticket/1362
>> but I still feel strongly that there is misunderstanding
>> and the bug/defect is valid. I would appreciate if someone would waste
>> more of his time to persuade me that I am wrong but please first read
>> till the end:
>>
>> The issue, as originally reported, is demonstrated with:
>>
>> ,---
>> | > python -c 'import numpy as N; print N.__version__; a=N.array([1,
>> (0,1)],dtype=object); print a==1; print a == (0,1), a[1] == (0,1)'
>> | 1.5.0.dev
>> | [ True False]
>> | [False False] True
>> `---
>>
>> whenever I expected the last line to be
>>
>> [False True] True
>>
>> charris (thanks for all the efforts to enlighten me) summarized it as
>>
>> """the result was correct given that the tuple (0,1) was converted to an
>> object array with elements 0 and 1. It is *not* converted to an array
>> containing a tuple. """
>>
>> and I was trying to argue that it is not the case in my example. It is
>> the case in charris's example though whenever both elements are of
>> the same length, or there is just a single tuple, i.e.
>>
>>
>
> The "problem" is that the tuple is converted to an array in the
> statement that
> does the comparison, not in the construction of the array. Numpy attempts
> to convert the right hand side of the == operator into an array. It
> then does
> the comparison using the two arrays.
>
> One way to get what you want is to create your own array and then do
> the comparison:
>
> In [1]: import numpy as np
>
> In [2]: a = np.array([1, (0,1)], dtype='O')
>
> In [3]: t = np.empty(1, dtype='O')
>
> In [4]: t[0] = (0,1)
>
> In [5]: a == t
> Out[5]: array([False, True], dtype=bool)
>
>
> In the above code, a numpy array 't' of objects with shape (1,) is created,
> and the single element is assigned the value (0,1). Then the comparison
> works as expected.
>
> More food for thought:
>
> In [6]: b = np.array([1, (0,1), "foo"], dtype='O')
>
> In [7]: b == 1
> Out[7]: array([ True, False, False], dtype=bool)
>
> In [8]: b == (0,1)
> Out[8]: False
>
> In [9]: b == "foo"
> Out[9]: array([False, False, True], dtype=bool)
>
It looks difficult to construct an object array with only 1 element,
since a tuple is interpreted as different array elements.
>>> N.array([(0,1)],dtype=object).shape
(1, 2)
>>> N.array([(0,1),()],dtype=object).shape
(2,)
>>> c = N.array([(0,1),()],dtype=object)[:1]
>>> c.shape1,)
>>> a == c
array([False, True], dtype=bool)
It looks like some convention is necessary for interpreting a tuple in
the array construction, but it doesn't look like a problem with the
comparison operator just a consequence.
Josef
> Warren
>
>> ,---
>> | In [1]: array((0,1), dtype=object)
>> | Out[1]: array([0, 1], dtype=object)
>> |
>> | In [2]: array((0,1), dtype=object).shape
>> | Out[2]: (2,)
>> `---
>>
>> There I would not expect my comparison to be valid indeed. But lets see what
>> happens in my case:
>>
>> ,---
>> | In [2]: array([1, (0,1)],dtype=object)
>> | Out[2]: array([1, (0, 1)], dtype=object)
>> |
>> | *In [3]: array([1, (0,1)],dtype=object).shape
>> | Out[3]: (2,)
>> |
>> | *In [4]: array([1, (0,1)],dtype=object)[1].shape
>> | ---
>> | AttributeError Traceback (most recent call
>> | last)
>> |
>> | /home/yoh/proj/ in ()
>> |
>> | AttributeError: 'tuple' object has no attribute 'shape'
>> `---
>>
>> So, as far as I see it, the array does contain an object of type tuple,
>> which does not get correctly compared upon __eq__ operation. Am I
>> wrong? Or does numpy internally somehow does convert 1st item (ie
>> tuple) into an array, but casts it back to tuple upon __repr__ or
>> __getitem__?
>>
>> Thanks in advance for feedback
>>
>> On Thu, 14 Jan 2010, NumPy Trac wrote:
>>
>>
>>> #1362: comparison operators (e.g. ==) on array with dtype object do not work
>>> -+--
>>> Reporter: yarikoptic | Owner: somebody
>>> Type: defect | Status: closed
>>> Priority: normal | Milestone:
>>> Component: Other | Version:
>>> Resolution: invalid | Keywords:
>>> -+--
>>> Changes (by charris):
>>>
>>
>>
>>> * status: reopened => closed
>>> * resolution: => invalid
>>>
>>
>>
>>
>>> Old description:
>>>
>>
>>
You can see this better with the '*' operator:
>>
>>
>>
{{{
In [8]: a * (0,2)
Out[8]: array([0, (0, 1, 0, 1)], dtype=object)
}}}
>>
>>
>>
Note how the tuple is concatenated with itself. The reason the original
instance of a worked was that 1 and (0,1) are of different lengths, so
the decent int
Re: [Numpy-discussion] comparison operators (e.g. ==) on array with dtype object do not work
Yaroslav Halchenko wrote:
> Dear NumPy People,
>
> First I want to apologize if I misbehaved on NumPy Trac by reopening the
> closed ticket
> http://projects.scipy.org/numpy/ticket/1362
> but I still feel strongly that there is misunderstanding
> and the bug/defect is valid. I would appreciate if someone would waste
> more of his time to persuade me that I am wrong but please first read
> till the end:
>
> The issue, as originally reported, is demonstrated with:
>
> ,---
> | > python -c 'import numpy as N; print N.__version__; a=N.array([1,
> (0,1)],dtype=object); print a==1; print a == (0,1), a[1] == (0,1)'
> | 1.5.0.dev
> | [ True False]
> | [False False] True
> `---
>
> whenever I expected the last line to be
>
> [False True] True
>
> charris (thanks for all the efforts to enlighten me) summarized it as
>
> """the result was correct given that the tuple (0,1) was converted to an
> object array with elements 0 and 1. It is *not* converted to an array
> containing a tuple. """
>
> and I was trying to argue that it is not the case in my example. It is
> the case in charris's example though whenever both elements are of
> the same length, or there is just a single tuple, i.e.
>
>
The "problem" is that the tuple is converted to an array in the
statement that
does the comparison, not in the construction of the array. Numpy attempts
to convert the right hand side of the == operator into an array. It
then does
the comparison using the two arrays.
One way to get what you want is to create your own array and then do
the comparison:
In [1]: import numpy as np
In [2]: a = np.array([1, (0,1)], dtype='O')
In [3]: t = np.empty(1, dtype='O')
In [4]: t[0] = (0,1)
In [5]: a == t
Out[5]: array([False, True], dtype=bool)
In the above code, a numpy array 't' of objects with shape (1,) is created,
and the single element is assigned the value (0,1). Then the comparison
works as expected.
More food for thought:
In [6]: b = np.array([1, (0,1), "foo"], dtype='O')
In [7]: b == 1
Out[7]: array([ True, False, False], dtype=bool)
In [8]: b == (0,1)
Out[8]: False
In [9]: b == "foo"
Out[9]: array([False, False, True], dtype=bool)
Warren
> ,---
> | In [1]: array((0,1), dtype=object)
> | Out[1]: array([0, 1], dtype=object)
> |
> | In [2]: array((0,1), dtype=object).shape
> | Out[2]: (2,)
> `---
>
> There I would not expect my comparison to be valid indeed. But lets see what
> happens in my case:
>
> ,---
> | In [2]: array([1, (0,1)],dtype=object)
> | Out[2]: array([1, (0, 1)], dtype=object)
> |
> | *In [3]: array([1, (0,1)],dtype=object).shape
> | Out[3]: (2,)
> |
> | *In [4]: array([1, (0,1)],dtype=object)[1].shape
> | ---
> | AttributeErrorTraceback (most recent call
> | last)
> |
> | /home/yoh/proj/ in ()
> |
> | AttributeError: 'tuple' object has no attribute 'shape'
> `---
>
> So, as far as I see it, the array does contain an object of type tuple,
> which does not get correctly compared upon __eq__ operation. Am I
> wrong? Or does numpy internally somehow does convert 1st item (ie
> tuple) into an array, but casts it back to tuple upon __repr__ or
> __getitem__?
>
> Thanks in advance for feedback
>
> On Thu, 14 Jan 2010, NumPy Trac wrote:
>
>
>> #1362: comparison operators (e.g. ==) on array with dtype object do not work
>> -+--
>> Reporter: yarikoptic | Owner: somebody
>> Type: defect | Status: closed
>> Priority: normal | Milestone:
>> Component: Other | Version:
>> Resolution: invalid |Keywords:
>> -+--
>> Changes (by charris):
>>
>
>
>> * status: reopened => closed
>> * resolution: => invalid
>>
>
>
>
>> Old description:
>>
>
>
>>> You can see this better with the '*' operator:
>>>
>
>
>
>>> {{{
>>> In [8]: a * (0,2)
>>> Out[8]: array([0, (0, 1, 0, 1)], dtype=object)
>>> }}}
>>>
>
>
>
>>> Note how the tuple is concatenated with itself. The reason the original
>>> instance of a worked was that 1 and (0,1) are of different lengths, so
>>> the decent into the nested sequence types stopped at one level and a
>>> tuple is one of the elements. When you do something like ((0,1),(0,1))
>>> the decent goes down two levels and you end up with a 2x2 array of
>>> integer objects. The rule of thumb for object arrays is that you get an
>>> array with as many indices as possible. Which is why object arrays are
>>> hard to create. Another example:
>>>
>
>
>
>>> {{{
>>> In [10]: array([(1,2,3),(1,2)], dtype=object)
>>> Out[10]: array([(1, 2, 3), (1, 2)], dtype=object)
>>>
>
>
>>> In [11]: array([(1,2),(1,2)], dtype=object)
>>> Out[11]:
>>> array([[1, 2],
>>>[1, 2]], dtype=object)
>>> }}}
>>>
>
[Numpy-discussion] comparison operators (e.g. ==) on array with dtype object do not work
Dear NumPy People,
First I want to apologize if I misbehaved on NumPy Trac by reopening the
closed ticket
http://projects.scipy.org/numpy/ticket/1362
but I still feel strongly that there is misunderstanding
and the bug/defect is valid. I would appreciate if someone would waste
more of his time to persuade me that I am wrong but please first read
till the end:
The issue, as originally reported, is demonstrated with:
,---
| > python -c 'import numpy as N; print N.__version__; a=N.array([1,
(0,1)],dtype=object); print a==1; print a == (0,1), a[1] == (0,1)'
| 1.5.0.dev
| [ True False]
| [False False] True
`---
whenever I expected the last line to be
[False True] True
charris (thanks for all the efforts to enlighten me) summarized it as
"""the result was correct given that the tuple (0,1) was converted to an
object array with elements 0 and 1. It is *not* converted to an array
containing a tuple. """
and I was trying to argue that it is not the case in my example. It is
the case in charris's example though whenever both elements are of
the same length, or there is just a single tuple, i.e.
,---
| In [1]: array((0,1), dtype=object)
| Out[1]: array([0, 1], dtype=object)
|
| In [2]: array((0,1), dtype=object).shape
| Out[2]: (2,)
`---
There I would not expect my comparison to be valid indeed. But lets see what
happens in my case:
,---
| In [2]: array([1, (0,1)],dtype=object)
| Out[2]: array([1, (0, 1)], dtype=object)
|
| *In [3]: array([1, (0,1)],dtype=object).shape
| Out[3]: (2,)
|
| *In [4]: array([1, (0,1)],dtype=object)[1].shape
| ---
| AttributeErrorTraceback (most recent call
| last)
|
| /home/yoh/proj/ in ()
|
| AttributeError: 'tuple' object has no attribute 'shape'
`---
So, as far as I see it, the array does contain an object of type tuple,
which does not get correctly compared upon __eq__ operation. Am I
wrong? Or does numpy internally somehow does convert 1st item (ie
tuple) into an array, but casts it back to tuple upon __repr__ or
__getitem__?
Thanks in advance for feedback
On Thu, 14 Jan 2010, NumPy Trac wrote:
> #1362: comparison operators (e.g. ==) on array with dtype object do not work
> -+--
> Reporter: yarikoptic | Owner: somebody
> Type: defect | Status: closed
> Priority: normal | Milestone:
> Component: Other | Version:
> Resolution: invalid |Keywords:
> -+--
> Changes (by charris):
> * status: reopened => closed
> * resolution: => invalid
> Old description:
> > You can see this better with the '*' operator:
> > {{{
> > In [8]: a * (0,2)
> > Out[8]: array([0, (0, 1, 0, 1)], dtype=object)
> > }}}
> > Note how the tuple is concatenated with itself. The reason the original
> > instance of a worked was that 1 and (0,1) are of different lengths, so
> > the decent into the nested sequence types stopped at one level and a
> > tuple is one of the elements. When you do something like ((0,1),(0,1))
> > the decent goes down two levels and you end up with a 2x2 array of
> > integer objects. The rule of thumb for object arrays is that you get an
> > array with as many indices as possible. Which is why object arrays are
> > hard to create. Another example:
> > {{{
> > In [10]: array([(1,2,3),(1,2)], dtype=object)
> > Out[10]: array([(1, 2, 3), (1, 2)], dtype=object)
> > In [11]: array([(1,2),(1,2)], dtype=object)
> > Out[11]:
> > array([[1, 2],
> >[1, 2]], dtype=object)
> > }}}
> New description:
> {{{
> python -c 'import numpy as N; print N.__version__; a=N.array([1,
> (0,1)],dtype=object); print a==1; print a == (0,1), a[1] == (0,1)'
> }}}
> results in
> {{{
> 1.5.0.dev
> [ True False]
> [False False] True
> }}}
> I expected last line to be
> {{{
> [False True] True
> }}}
> So, it works for int but doesn't work for tuple... I guess it doesn't try
> to compare element by element but does smth else.
--
Yaroslav O. Halchenko
Postdoctoral Fellow, Department of Psychological and Brain Sciences
Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
Phone: +1 (603) 646-9834 Fax: +1 (603) 646-1419
WWW: http://www.linkedin.com/in/yarik
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] hello~
Hi,One of my friends introduce a very good website to me: http://www.flsso.com/. All their products are new and original. They have many brands, such as Sony, HP, Apple, Nokia and so on. Now , they are promoting their products for the coustomers. So their prices are very competitive. By the way, they mainly sell iphones, laptops, tvs, playstation and so on.If you need these products, it will be a good choice.Regards! ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] dtype.isbuiltin changed by .newbyteorder
Hi, > It is at least undesirable. It may not be a bug per se as I don't > think that we guarantee that .isbuiltin is free from false negatives > (though we do guarantee that it is free from false positives). The > reason is that we would have to search the builtin dtypes for a match > every time we create a new dtype object, and that could be more > expensive than we care to do for *every* creation of a dtype object. > It is possible that we can have a cheaper heuristic (native byte order > and the standard typecodes) and that transformations like > .newbyteorder() can have just a teeny bit more intelligent logic about > how it transforms the .isbuiltin flag. Thanks - that's very clear and helpful, and made me realize I didn't understand the builtin attribute. I suppose something like the following: output_dtype.isbuiltin = input_dtype.isbuiltin and new_byteorder == native would at least reduce the false negatives at little cost. Cheers, Matthew ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] dtype.isbuiltin changed by .newbyteorder
On Thu, Jan 14, 2010 at 07:01, Matthew Brett wrote:
> Hi,
>
> Over on the scipy list, someone pointed out an oddness in the output
> of the matlab reader, which revealed this - to me - unexpected
> behavior in numpy:
>
> In [20]: dt = np.dtype('f8')
>
> In [21]: dt.isbuiltin
> Out[21]: 1
>
> In [22]: ndt = dt.newbyteorder('<')
>
> In [23]: ndt.isbuiltin
> Out[23]: 0
>
> I was expecting the 'isbuiltin' attribute to be the same (1) after
> byte swapping. Does that seem reasonable to y'all? Then, is this a
> bug?
It is at least undesirable. It may not be a bug per se as I don't
think that we guarantee that .isbuiltin is free from false negatives
(though we do guarantee that it is free from false positives). The
reason is that we would have to search the builtin dtypes for a match
every time we create a new dtype object, and that could be more
expensive than we care to do for *every* creation of a dtype object.
It is possible that we can have a cheaper heuristic (native byte order
and the standard typecodes) and that transformations like
.newbyteorder() can have just a teeny bit more intelligent logic about
how it transforms the .isbuiltin flag.
Just for clarity and future googling, the issue is that when a native
dtype has .newbyteorder() called on it to make a new dtype that has
the *same* native byte order, the .isbuiltin flag incorrectly states
that it is not builtin. Using .newbyteorder() to swap the byte order
to the non-native byte order should and does cause the resulting dtype
to not be considered builtin.
--
Robert Kern
"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
-- Umberto Eco
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] dtype.isbuiltin changed by .newbyteorder
Hi,
Over on the scipy list, someone pointed out an oddness in the output
of the matlab reader, which revealed this - to me - unexpected
behavior in numpy:
In [20]: dt = np.dtype('f8')
In [21]: dt.isbuiltin
Out[21]: 1
In [22]: ndt = dt.newbyteorder('<')
In [23]: ndt.isbuiltin
Out[23]: 0
I was expecting the 'isbuiltin' attribute to be the same (1) after
byte swapping.Does that seem reasonable to y'all? Then, is this a
bug?
Thanks a lot,
Matthew
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Matrix vs array in ma.minimum
On Jan 14, 2010, at 4:53 AM, David Cournapeau wrote: > Hi, > > I encountered a problem in matlab which boils down to a surprising > behavior of np.ma.minimum: > > x = np.random.randn(2, 3) > mx = np.matrix(x) > > np.ma.minimum(x) # smallest item of x > ret = np.ma.minimum(mx) # flattened version of mx, i.e. ret == mx.flatten() > > Is this expected ? Er, no. np.ma.minimum(a, b) returns the lowest value of a and b element-wsie, or the the lowest element of a is b is None. The behavior is inherited from the very first implementation of maskedarray in numeric. This itself is unexpected, since np.minimum requires at least 2 input arguments. As you observed, the current function breaks down w/ np.matrix objects when only one argument is given (and when the axis is None): we call umath.minimum.reduce on the ravelled matirx, which returns the ravelled matrix. One would expect a scalar, so yes, this behavior is also unexpected. Now, which way should we go ? Keep np.ma.minimum as it is (fixing the bug so that a scalar is returned if the function is called with only 1 argument and an axis None) ? Adapt it to match np.minimum ? ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] Matrix vs array in ma.minimum
Hi, I encountered a problem in matlab which boils down to a surprising behavior of np.ma.minimum: x = np.random.randn(2, 3) mx = np.matrix(x) np.ma.minimum(x) # smallest item of x ret = np.ma.minimum(mx) # flattened version of mx, i.e. ret == mx.flatten() Is this expected ? cheers, David ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] wrong casting of augmented assignment statements
I've written a self-contained example that shows that numpy indeed
tries to call the __float__ method.
What is buggy is what happens if calling the __float__ method raises
an Exception.
Then numpy assumes (in this case wrongly) that the object should be
casted to the neutral element.
I'd guess that the __float__ method is called somewhere in a try:
statement and if an exception is raised it is casted to the neutral
element.
I've tried to locate the corresponding code in the numpy sources but I
got lost. Could someone be so kind and point me to it?
start code --
import numpy
print 'numpy.__version__ = ',numpy.__version__
class ad1:
def __init__(self,x):
self.x = x
def __mul__(self,other):
if not isinstance(other, self.__class__):
return self.__class__(self.x * other)
return self.__class__(self.x * other.x)
def __rmul__(self,other):
return self * other
def __float__(self):
raise Exception('this is not possible')
def __str__(self):
return str(self.x)
print '\nThis example yields buggy behavior:'
x1 = numpy.array([ad1(1.), ad1(2.), ad1(3.)])
y1 = numpy.random.rand(3)
print 'y1= ',y1
print 'x1= ',x1
z1 = x1 * y1
y1 *= x1# this should call the __float__ method of ad1
which would raise an Exception
print 'z1=x1*y1',z1
print 'y1*=x1 ',y1
class ad2:
def __init__(self,x):
self.x = x
def __mul__(self,other):
if not isinstance(other, self.__class__):
return self.__class__(self.x * other)
return self.__class__(self.x * other.x)
def __rmul__(self,other):
return self * other
def __float__(self):
return float(self.x)
def __str__(self):
return str(self.x)
print '\nThis example works fine:'
x2 = numpy.array([ad2(1.), ad2(2.), ad2(3.)])
y2 = numpy.random.rand(3)
print 'y2= ',y2
print 'x2= ',x2
z2 = x2 * y2
y2 *= x2# this should call the __float__ method of ad1
which would raise an Exception
print 'z2=x2*y2',z2
print 'y2*=x2 ',y2
end code --
output -
wal...@wronski$ python
wrong_casting_object_to_float_of_augmented_assignment_statements.py
numpy.__version__ = 1.3.0
This example yields buggy behavior:
y1= [ 0.15322371 0.47915903 0.81153995]
x1= [1.0 2.0 3.0]
z1=x1*y1 [0.153223711127 0.958318053803 2.43461983729]
y1*=x1 [ 0.15322371 0.47915903 0.81153995]
This example works fine:
y2= [ 0.49377037 0.60908423 0.79772095]
x2= [1.0 2.0 3.0]
z2=x2*y2 [0.493770370747 1.21816846399 2.39316283707]
y2*=x2 [ 0.49377037 1.21816846 2.39316284]
end output -
On Tue, Jan 12, 2010 at 7:38 PM, Robert Kern wrote:
> On Tue, Jan 12, 2010 at 12:31, Sebastian Walter
> wrote:
>> On Tue, Jan 12, 2010 at 7:09 PM, Robert Kern wrote:
>>> On Tue, Jan 12, 2010 at 12:05, Sebastian Walter
>>> wrote:
Hello,
I have a question about the augmented assignment statements *=, +=, etc.
Apparently, the casting of types is not working correctly. Is this
known resp. intended behavior of numpy?
>>>
>>> Augmented assignment modifies numpy arrays in-place, so the usual
>>> casting rules for assignment into an array apply. Namely, the array
>>> being assigned into keeps its dtype.
>>
>> what are the usual casting rules?
>
> For assignment into an array, the array keeps its dtype and the data
> being assigned into it will be cast to that dtype.
>
>> How does numpy know how to cast an object to a float?
>
> For a general object, numpy will call its __float__ method.
>
>>> If you do not want in-place modification, do not use augmented assignment.
>>
>> Normally, I'd be perfectly fine with that.
>> However, this particular problem occurs when you try to automatically
>> differentiate an algorithm by using an Algorithmic Differentiation
>> (AD) tool.
>> E.g. given a function
>>
>> x = numpy.ones(2)
>> def f(x):
>> a = numpy.ones(2)
>> a *= x
>> return numpy.sum(a)
>>
>> one would use an AD tool as follows:
>> x = numpy.array([adouble(1.), adouble(1.)])
>> y = f(x)
>>
>> but since the casting from object to float is not possible the
>> computed gradient \nabla_x f(x) will be wrong.
>
> Sorry, but that's just a limitation of the AD approach. There are all
> kinds of numpy constructions that AD can't handle.
>
> --
> Robert Kern
>
> "I have come to believe that the whole world is an enigma, a harmless
> enigma that is made terrible by our own mad attempt to interpret it as
> though it had an underlying truth."
> -- Umberto Eco
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Getting Callbacks with arrays to work
Hi, Thanks all works now! The implicit none only didn't work when defining dv as a function now its a subroutine it seems to work. Regards Jon On 12/01/2010 13:44, Pearu Peterson wrote: > Hi, > > The problem is that f2py does not support callbacks that > return arrays. There is easy workaround to that: provide > returnable arrays as arguments to callback functions. > Using your example: > > SUBROUTINE CallbackTest(dv,v0,Vout,N) >IMPLICIT NONE > >!F2PY intent( hide ):: N >INTEGER:: N, ic >EXTERNAL:: dv > >DOUBLE PRECISION, DIMENSION( N ), INTENT(IN):: v0 >DOUBLE PRECISION, DIMENSION( N ), INTENT(OUT):: Vout > >DOUBLE PRECISION, DIMENSION( N ):: Vnow >DOUBLE PRECISION, DIMENSION( N ):: temp > >Vnow = v0 >!f2py intent (out) temp >call dv(temp, Vnow, N) > >DO ic = 1, N > Vout( ic ) = temp(ic) >END DO > > END SUBROUTINE CallbackTest > > $ f2py -c test.f90 -m t --fcompiler=gnu95 > from numpy import * from t import * arr = array([2.0, 4.0, 6.0, 8.0]) def dV(v): > print 'in Python dV: V is: ',v > ret = v.copy() > ret[1] = 100.0 > return ret > ... output = callbacktest(dV, arr) > in Python dV: V is: [ 2. 4. 6. 8.] output > array([ 2., 100.,6.,8.]) > > What problems do you have with implicit none? It works > fine here. Check the format of your source code, > if it is free then use `.f90` extension, not `.f`. > > HTH, > Pearu > > Jon Moore wrote: >> Hi, >> >> I'm trying to build a differential equation integrator and later a >> stochastic differential equation integrator. >> >> I'm having trouble getting f2py to work where the callback itself >> receives an array from the Fortran routine does some work on it and then >> passes an array back. >> >> For the stoachastic integrator I'll need 2 callbacks both dealing with >> arrays. >> >> The idea is the code that never changes (ie the integrator) will be in >> Fortran and the code that changes (ie the callbacks defining >> differential equations) will be different for each problem. >> >> To test the idea I've written basic code which should pass an array back >> and forth between Python and Fortran if it works right. >> >> Here is some code which doesn't work properly:- >> >> SUBROUTINE CallbackTest(dv,v0,Vout,N) >> !IMPLICIT NONE >> >> cF2PY intent( hide ):: N >> INTEGER:: N, ic >> >> EXTERNAL:: dv >> >> DOUBLE PRECISION, DIMENSION( N ), INTENT(IN):: v0 >> DOUBLE PRECISION, DIMENSION( N ), INTENT(OUT):: Vout >> >> DOUBLE PRECISION, DIMENSION( N ):: Vnow >> DOUBLE PRECISION, DIMENSION( N ):: temp >> >> Vnow = v0 >> >> >> temp = dv(Vnow, N) >> >> DO ic = 1, N >> Vout( ic ) = temp(ic) >> END DO >> >> END SUBROUTINE CallbackTest >> >> >> >> When I test it with this python code I find the code just replicates the >> first term of the array! >> >> >> >> >> from numpy import * >> import callback as c >> >> def dV(v): >> print 'in Python dV: V is: ',v >> return v.copy() >> >> arr = array([2.0, 4.0, 6.0, 8.0]) >> >> print 'Arr is: ', arr >> >> output = c.CallbackTest(dV, arr) >> >> print 'Out is: ', output >> >> >> >> >> Arr is: [ 2. 4. 6. 8.] >> >> in Python dV: V is: [ 2. 4. 6. 8.] >> >> Out is: [ 2. 2. 2. 2.] >> >> >> >> Any ideas how I should do this, and also how do I get the code to work >> with implicit none not commented out? >> >> Thanks >> >> Jon >> >> >> >> >> >> ___ >> NumPy-Discussion mailing list >> NumPy-Discussion@scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion > ___ > NumPy-Discussion mailing list > NumPy-Discussion@scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
