Re: [Numpy-discussion] request for SWIG numpy.i users

[Tom Krauss]

Hi folks,

I just downloaded Bill's numpy.i
at
commit 4dcb0679, and tried it out a bit on some of my personal projects.
 It worked fine, although I use only a fraction of the capabilities that it
includes.

 And, it made the warning go away!

I used to get this warning

g++ -g -fPIC -c simple_wrap.cpp -I/usr/include/python2.7
-I/Users/tkrauss/projects/dev_env/lib/python2.7/site-packages/numpy-1.8.0.dev_f2f0ac0_20120725-py2.7-macosx-10.8-x86_64.egg/numpy/core/include
In file included from
/Users/tkrauss/projects/dev_env/lib/python2.7/site-packages/numpy-1.8.0.dev_f2f0ac0_20120725-py2.7-macosx-10.8-x86_64.egg/numpy/core/include/numpy/ndarraytypes.h:1722,
 from
/Users/tkrauss/projects/dev_env/lib/python2.7/site-packages/numpy-1.8.0.dev_f2f0ac0_20120725-py2.7-macosx-10.8-x86_64.egg/numpy/core/include/numpy/ndarrayobject.h:17,
 from
/Users/tkrauss/projects/dev_env/lib/python2.7/site-packages/numpy-1.8.0.dev_f2f0ac0_20120725-py2.7-macosx-10.8-x86_64.egg/numpy/core/include/numpy/arrayobject.h:15,
 from simple_wrap.cpp:3062:
/Users/tkrauss/projects/dev_env/lib/python2.7/site-packages/numpy-1.8.0.dev_f2f0ac0_20120725-py2.7-macosx-10.8-x86_64.egg/numpy/core/include/numpy/npy_deprecated_api.h:11:2:
warning: #warning "Using deprecated NumPy API, disable it by #defining
NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION"

but not with this version.

You can see which version of numpy I am using there, and that I am on Mac
OS X 10.8. (10.8.4 specifically)  Python 2.7.2

I'd say SHIP IT!

Nice work, thanks for all your work on numpy and numpy.i.

- Tom Krauss



On Tue, Jun 4, 2013 at 3:13 PM, Ralf Gommers  wrote:

> Hi,
>
> If you're using or are very familiar with SWIG and the numpy.i interface
> to it, please help to test and/or review
> https://github.com/numpy/numpy/pull/3148. It's a fairly major update to
> numpy.i by Bill Spotz, containing the following:
>   - support for 4D arrays and memory managed output arguments
>   - rework for the deprecated API's in numpy 1.6 and 1.7
>   - a bug fix in a 3D typemap
>   - documentation improvements
>
> It would be good to have this merged before branching 1.8.x. Not many of
> the regular reviewers of numpy PRs are familiar with numpy.i, therefore
> help would be much appreciated.
>
> Thanks,
> Ralf
>
>
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] np.asarray atleast_float64 ?

[josef . pktd]

On Sat, Jun 8, 2013 at 7:05 PM, Sebastian Berg
 wrote:
> On Sat, 2013-06-08 at 08:52 -0400, josef.p...@gmail.com wrote:
>> Is there anything to require a numpy array with a minimum numeric dtype?
>>
>> To avoid lower precision calculations and be upwards compatible, something 
>> like
>>
>> x = np.asarray(x, >=np.float64)
>
> np.result_type(arr, np.float64) uses the usual numpy promotion rules.
> But it doesn't do the "asarray" part. Its still the closest thing I can
> think of right now.

Thank you, that looks close enough

And I will soon switch to numpy 1.6 and can read up on some old What's new.

Josef

>
> - Sebastian
>
>>
>> that converts ints, bool and lower precision to float64 but leaves
>> higher precision float and complex double alone.
>>
>>
>> Josef
>> ___
>> NumPy-Discussion mailing list
>> NumPy-Discussion@scipy.org
>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>
>
>
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] np.asarray atleast_float64 ?

[Sebastian Berg]

On Sat, 2013-06-08 at 08:52 -0400, josef.p...@gmail.com wrote:
> Is there anything to require a numpy array with a minimum numeric dtype?
> 
> To avoid lower precision calculations and be upwards compatible, something 
> like
> 
> x = np.asarray(x, >=np.float64)

np.result_type(arr, np.float64) uses the usual numpy promotion rules.
But it doesn't do the "asarray" part. Its still the closest thing I can
think of right now.

- Sebastian

> 
> that converts ints, bool and lower precision to float64 but leaves
> higher precision float and complex double alone.
> 
> 
> Josef
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
> 


___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] adding booleans

[josef . pktd]

On Sat, Jun 8, 2013 at 7:54 AM, Robert Kern  wrote:
> On Sat, Jun 8, 2013 at 12:40 PM,   wrote:
>> Question about namespace
>>
>> why is there bool and bool_ ?
>>
> np.bool(True) + np.bool(True)
>> 2
> np.bool_(True) + np.bool_(True)
>> True
>>
> type(np.bool(True))
>> 
> type(np.bool_(True))
>> 
>>
>> I didn't pay attention to the trailing underline in Pauli's original example
>
> `np.bool is __builtin__.bool`. It's a backwards-compatibility alias
> for an old version of numpy that used the name `np.bool` for the
> scalar type now called `np.bool_` (similarly for `np.float`, `np.int`,
> and `np.complex`). Since `from numpy import *` caused name collisions,
> we moved the scalar types to their current underscored versions.
> However, since people had started to use `dtype=np.bool`, we just
> aliased the builtin type objects to keep that code working. I would
> love to be able to remove those since they are an unending source of
> confusion. However, there is no good way to issue deprecation warnings
> when you use them, so that means that upgrades would cause sudden,
> unwarned breakage, which we like to avoid.

Thanks for the explanation.

I guess I never ran into a problem with this, and never saw the difference.

It's confusing, but if the behavior is the same (in all usual cases)
whether it's python or numpy, then it is less of a problem.

>>> np.int64(0) ** np.int64(-1)
inf
>>> np.int(0) ** np.int(-1)
Traceback (most recent call last):
  File "", line 1, in 
np.int(0) ** np.int(-1)
ZeroDivisionError: 0.0 cannot be raised to a negative power

Another thing to watch out for.

Josef

>
> --
> Robert Kern
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

[Numpy-discussion] np.asarray atleast_float64 ?

[josef . pktd]

Is there anything to require a numpy array with a minimum numeric dtype?

To avoid lower precision calculations and be upwards compatible, something like

x = np.asarray(x, >=np.float64)

that converts ints, bool and lower precision to float64 but leaves
higher precision float and complex double alone.


Josef
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] adding booleans

[josef . pktd]

On Sat, Jun 8, 2013 at 5:18 AM, Sebastian Berg
 wrote:
> On Sat, 2013-06-08 at 00:48 +0100, Nathaniel Smith wrote:
>> On 7 Jun 2013 21:58,  wrote:
>> >
>> > Interesting observation, (while lurking on a pull request)
>> >
>> > >>> np.add.reduce(np.arange(5)<3)
>> > 3
>> > >>> np.add((np.arange(5)<3), (np.arange(5)<3))
>> > array([ True,  True,  True, False, False], dtype=bool)
>> >
>> >
>> > I often use summing of an array of boolean but didn't know the
>> second behavior
>>
>> ...yeah weird. My gut reaction is that it's a bug. Addition on bools
>> should either be an error, undefined but doable via an implicit upcast
>> to int (analogous to calling np.sin on an int array triggering an
>> upcast to float), or xor (i.e., addition mod 2). But apparently we're
>> inconsistent - add.reduce upcasts, and add.__call__, uh... upcasts and
>> then downcasts, maybe? It's like if np.sin on an int array returned
>> ints? I can't see how to get the quoted behaviour in any conceptually
>> coherent way. But maybe I'm missing something.
>>
> I believe these are implemented as ufunc-machinery special cases for the
> add ufunc reduction methods. This ensures that the sums output integer
> type is at least a `np.int_` if the input is integer or bool. With a
> future warning, it seems plausible for bools to have `bool + bool = int`
> (code might (ab)use it as a logical operation) I guess, but `int16 +
> int16` should probably stay int16? While sum over it being larger avoids
> overflow accidents.

I think bool + bool with overflow result would be consistent with the rest.
if this is the rule
"reduce/accumulate operations upcast, simple operators on 2 identical
types don't"

but bool + bool has a result that is neither python nor overflow, AFAICS

I never looked carefully at what the expected overflow behavior is
with "small" dtypes (since I avoid them)

In python 3, division is now the odd case

>>> np.uint8(1) - np.uint8(2)

Warning (from warnings module):
  File "C:\Programs\Python32\Lib\idlelib\idle.pyw", line 1
try:
RuntimeWarning: overflow encountered in ubyte_scalars
>>> 255
>>> np.uint8(1) / np.uint8(2)
0.5
>>> np.uint8(1) * np.uint8(2)
2
>>> np.uint8(150) * np.uint8(2)
44



Josef

> Another (additionally good) thing would be to have better integer
> overflow warnings here, but it seems likely that for the sake of
> backward compatibility these special cases can't be easily changed in
> any case.
>
> - Sebastian
>
>> -n
>>
>> ___
>> NumPy-Discussion mailing list
>> NumPy-Discussion@scipy.org
>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] supporting quad precision

[Anne Archibald]

Looking at the rational module, I think you're right: it really shouldn't
be too hard to get quads working as a user type using gcc's __float128
type, which will provide hardware arithmetic in the unlikely case that the
user has hardware quads. Alternatively, probably more work, one could use a
package like qd to provide portable quad precision (and quad-double).

I'll take a look.

Anne
On Jun 5, 2013 7:10 PM, "David Cournapeau"  wrote:

> On Wed, Jun 5, 2013 at 5:21 PM, Charles R Harris
>  wrote:
> > Hi Anne,
> >
> > Long time no see ;)
> >
> > On Wed, Jun 5, 2013 at 10:07 AM, Anne Archibald 
> wrote:
> >>
> >> Hi folks,
> >>
> >> I recently came across an application I needed quad precision for
> >> (high-accuracy solution of a differential equation). I found a C++
> library
> >> (odeint) that worked for the integration itself, but unfortunately it
> >> appears numpy is not able to work with quad precision arrays. For my
> >> application the quad precision is only really needed for integrator
> state,
> >> so I can manually convert my data to long doubles as I go to and from
> numpy,
> >> but it's a pain. So quad precision support would be nice.
> >>
> >> There's a thread discussing quad support:
> >>
> http://mail.scipy.org/pipermail/numpy-discussion/2012-February/061080.html
> >> Essentially, there isn't any, but since gcc >= 4.6 supports them on
> Intel
> >> hardware (in software), it should be possible. (Then the thread got
> bogged
> >> down in bike-shedding about what to call them.)
> >>
> >> What would it take to support quads in numpy? I looked into the numpy
> base
> >> dtype definitions, and it's a complex arrangement involving detection of
> >> platform support and templatized C code; in the end I couldn't really
> see
> >> where to start. But it seems to me all the basics are available: native
> C
> >> syntax for basic arithmetic, "qabs"-style versions of all the basic
> >> functions, NaNs and Infs. So how would one go about adding quad support?
> >>
> >
> > There are some improvements for user types committed in 1.8-dev. Perhaps
> > quad support could be added as a user type as it is still
> platform/compiler
> > dependent. The rational type added to numpy could supply a template for
> > adding the new type.
>
> I would be in support of that direction as well: let it live
> separately until CPU/compiler support is coming up.
>
> Anne, will you be at scipy conference ? Improving user data type
> internal API is something I'd like to work on as well
>
> David
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] adding booleans

[Robert Kern]

On Sat, Jun 8, 2013 at 12:40 PM,   wrote:
> Question about namespace
>
> why is there bool and bool_ ?
>
 np.bool(True) + np.bool(True)
> 2
 np.bool_(True) + np.bool_(True)
> True
>
 type(np.bool(True))
> 
 type(np.bool_(True))
> 
>
> I didn't pay attention to the trailing underline in Pauli's original example

`np.bool is __builtin__.bool`. It's a backwards-compatibility alias
for an old version of numpy that used the name `np.bool` for the
scalar type now called `np.bool_` (similarly for `np.float`, `np.int`,
and `np.complex`). Since `from numpy import *` caused name collisions,
we moved the scalar types to their current underscored versions.
However, since people had started to use `dtype=np.bool`, we just
aliased the builtin type objects to keep that code working. I would
love to be able to remove those since they are an unending source of
confusion. However, there is no good way to issue deprecation warnings
when you use them, so that means that upgrades would cause sudden,
unwarned breakage, which we like to avoid.

--
Robert Kern
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] adding booleans

[josef . pktd]

Question about namespace

why is there bool and bool_ ?

>>> np.bool(True) + np.bool(True)
2
>>> np.bool_(True) + np.bool_(True)
True

>>> type(np.bool(True))

>>> type(np.bool_(True))


I didn't pay attention to the trailing underline in Pauli's original example

Josef
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] adding booleans

[Sebastian Berg]

On Fri, 2013-06-07 at 20:29 -0400, josef.p...@gmail.com wrote:
> On Fri, Jun 7, 2013 at 8:08 PM,   wrote:
> > On Fri, Jun 7, 2013 at 7:48 PM, Nathaniel Smith  wrote:
> >> On 7 Jun 2013 21:58,  wrote:
> >>>
> >>> Interesting observation, (while lurking on a pull request)
> >>>
> >>> >>> np.add.reduce(np.arange(5)<3)
> >>> 3
> >>> >>> np.add((np.arange(5)<3), (np.arange(5)<3))
> >>> array([ True,  True,  True, False, False], dtype=bool)
> >>>
> >>>
> >>> I often use summing of an array of boolean but didn't know the second
> >>> behavior
> >>
> >> ...yeah weird. My gut reaction is that it's a bug. Addition on bools should
> >> either be an error, undefined but doable via an implicit upcast to int
> >> (analogous to calling np.sin on an int array triggering an upcast to 
> >> float),
> >> or xor (i.e., addition mod 2). But apparently we're inconsistent -
> >> add.reduce upcasts, and add.__call__, uh... upcasts and then downcasts,
> >> maybe? It's like if np.sin on an int array returned ints? I can't see how 
> >> to
> >> get the quoted behaviour in any conceptually coherent way. But maybe I'm
> >> missing something.
> >
> > The first case is perfectly good behavior. I always "knew"/assumed
> > that in python bool are 0-1 ints with all the calculation rules.
> > I only found the second one a surprise (found by Pauli)
> >
>  reduce(np.add, [ True,  True,  True, False, False])
> > True
>  reduce(lambda x, y: x+y, [ True,  True,  True, False, False])
> > 3
> >
> >
> > The following we use *very* often:
> >
> > proportion = (x > 0).mean()
> > n_valid = isfinite(x).sum()
> >
> > cond = cond1 * cond2
> >
> > in python: trick indexing with 0-1 bool
>  ["True", "False"][False]
> > 'True'
> 
> python indexing with np.bool
> >>> ["True", "False"][np.bool(False)]
> 'True'
> >>> ["True", "False"][np.bool(True)]
> 'False'
> 

Uhh, damnit, I thought this was not the case. The correct code would be
`np.bool_(False)`, but it works as well. I guess for the sake of
compatibility with Python bools it must work...
Within NumPy indexing with bools however seems wrong to me. This creates
a bit of dilemma in my mind, because I thought it would be fine to just
say that python bools cannot be safely interpreted as integers (i.e.
ignore that they support the index operator), but seeing this we would
have to check for `np.bool_` too.
Also note that `operator.index(np.array(True))` actually fails
currently, which is a (harmless) inconsistency.

It seems to mean that we will either have a 0-d special case (allow
0darr[np.bool_] as boolean indexing but otherwise use it as int), or
need extra checks for `np.bool_` to disallow both python and numpy
bools.

- Sebastian

> operation between numbers and bool
> >>> a = np.array([ True,  True,  True, False, False])
> >>> a * range(5)
> array([0, 1, 2, 0, 0])
> >>> a * range(1, 6)
> array([1, 2, 3, 0, 0])
> >>> a + range(5)
> array([1, 2, 3, 3, 4])
> 
> 
> multiply and maximum don't need to upcast because the result stays 0-1
> 
> >>> reduce(np.multiply, [ True,  True,  True, False, False])
> False
> >>> np.multiply.reduce([ True,  True,  True, False, False])
> 0
> 
> >>> np.maximum.reduce([ True,  True,  True, False, False])
> True
> >>> np.maximum.accumulate([ True,  True,  True, False, False])
> array([ True,  True,  True,  True,  True], dtype=bool)
> 
> also fine
> >>> np.add.accumulate([ True,  True,  True, False, False])
> array([1, 2, 3, 3, 3])
> 
> ---
> 
> I think the only "weird" and inconsistent one is that bool1 + bool2
> does not upcast to int
> 
> Josef
> 
> >
> > Josef
> >
> >>
> >> -n
> >>
> >>
> >> ___
> >> NumPy-Discussion mailing list
> >> NumPy-Discussion@scipy.org
> >> http://mail.scipy.org/mailman/listinfo/numpy-discussion
> >>
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
> 


___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] adding booleans

[Pauli Virtanen]

08.06.2013 02:48, Nathaniel Smith kirjoitti:
[clip]
> ...yeah weird. My gut reaction is that it's a bug. Addition on bools
> should either be an error, undefined but doable via an implicit upcast
> to int (analogous to calling np.sin on an int array triggering an upcast
> to float), or xor (i.e., addition mod 2).

There are several more sensible choices:

http://en.wikipedia.org/wiki/Boolean_ring

___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] adding booleans

[Sebastian Berg]

On Sat, 2013-06-08 at 00:48 +0100, Nathaniel Smith wrote:
> On 7 Jun 2013 21:58,  wrote:
> >
> > Interesting observation, (while lurking on a pull request)
> >
> > >>> np.add.reduce(np.arange(5)<3)
> > 3
> > >>> np.add((np.arange(5)<3), (np.arange(5)<3))
> > array([ True,  True,  True, False, False], dtype=bool)
> >
> >
> > I often use summing of an array of boolean but didn't know the
> second behavior
> 
> ...yeah weird. My gut reaction is that it's a bug. Addition on bools
> should either be an error, undefined but doable via an implicit upcast
> to int (analogous to calling np.sin on an int array triggering an
> upcast to float), or xor (i.e., addition mod 2). But apparently we're
> inconsistent - add.reduce upcasts, and add.__call__, uh... upcasts and
> then downcasts, maybe? It's like if np.sin on an int array returned
> ints? I can't see how to get the quoted behaviour in any conceptually
> coherent way. But maybe I'm missing something.
> 
I believe these are implemented as ufunc-machinery special cases for the
add ufunc reduction methods. This ensures that the sums output integer
type is at least a `np.int_` if the input is integer or bool. With a
future warning, it seems plausible for bools to have `bool + bool = int`
(code might (ab)use it as a logical operation) I guess, but `int16 +
int16` should probably stay int16? While sum over it being larger avoids
overflow accidents.
Another (additionally good) thing would be to have better integer
overflow warnings here, but it seems likely that for the sake of
backward compatibility these special cases can't be easily changed in
any case.

- Sebastian

> -n
> 
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion


___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion