Re: [Numpy-discussion] Please keep Numpy master working on Py3

[Charles R Harris]

On Thu, Feb 3, 2011 at 11:07 AM, Mark Wiebe  wrote:

> On Tue, Feb 1, 2011 at 10:09 PM, David  wrote:
>
>> On 02/02/2011 02:57 PM, Mark Wiebe wrote:
>> > On Tue, Feb 1, 2011 at 9:49 PM, David > > > wrote:
>> >
>> > 
>> >
>> >
>> >  > In the meantime, I put the relevant header in
>> > numpy/core/src/private, to
>> >  > make the dependency clearer.
>> >
>> > Following that argument, there are other unwanted dependencies
>> between
>> > multiarray and ufunc, causing circular dependencies. I don't think
>> they
>> > were there before, and it makes building numpy with a
>> dependency-based
>> > tool like scons or waf extremely difficult.
>> >
>> >
>> > This particular separation of two components felt somewhat artificial to
>> > me.  For the iterator buffering, as an example, I initially thought it
>> > would be good to use the same default as the ufuncs do.  Instead I ended
>> > up just using a hard-coded default.
>>
>> Not sure I understand the exact argument, but if it is just a matter of
>> getting default values, it is easy to share them using a common header.
>>
>
> I guess my argument was just that a few more ufunc features need to migrate
> to the core.  I don't think this needs to be done at the moment, though.
>
>
>>
>> > I think the error handling policy
>> > in the ufuncs could also be useful for computations in the core.
>>
>> At the moment, ufunc and multiarray are separate extensions. If each
>> depends on the other API-wise, it is an issue. If there are some
>> commonalities, then they can be put in a separate extensions. They are
>> already way too big as they are (historical reasons), we should really
>> fight this for maintanability.
>>
>> I realize the code organization and the build stuff is a bit of a mess
>> at the moment, so if you have questions on that aspect, feel free to ask
>> clarification or help,
>
>
> Yeah, I understand it's the result of organic growth and merging from many
> different sources. The core library should probably become layered in a
> manner roughly as follows, with each layer depending only on the previous
> APIs.  This is what Travis was getting at, I believe, with the generator
> array idea affecting mainly the Storage and Iteration APIs, generalizing
> them so that new storage and iteration methods can be plugged in.
>
> Data Type API: data type numbers, casting, byte-swapping, etc.
> Array Storage API: array creation/manipulation/deletion.
> Array Iteration API: array iterators, copying routines.
> Array Calculation API: typedefs for various types of calculation
> functions, common calculation idioms, ufunc creation API, etc.
>
> Then, the ufunc module would use the Array Calculation API to implement all
> the ufuncs and other routines like inner, dot, trace, diag, tensordot,
> einsum, etc.
>
>
I like the lower two levels if, as I assume, they are basically aimed at
allocating, deallocating blocks of memory (or equivalent) and doing basic
manipulations such as dealing with endianess and casting. Where do you see
array methods making an appearance? The original Numeric only had three
(IIRC) rather basic methods and everything else was function based, an
approach which is probably easier to maintain. The extensive use of methods
came from numarray and might be something that could be added at a higher
level so that the current ndarrays would be objects combining ow level
arrays and ufuncs.

Chuck
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Please keep Numpy master working on Py3

[Mark Wiebe]

On Tue, Feb 1, 2011 at 10:09 PM, David  wrote:

> On 02/02/2011 02:57 PM, Mark Wiebe wrote:
> > On Tue, Feb 1, 2011 at 9:49 PM, David  > > wrote:
> >
> > 
> >
> >
> >  > In the meantime, I put the relevant header in
> > numpy/core/src/private, to
> >  > make the dependency clearer.
> >
> > Following that argument, there are other unwanted dependencies
> between
> > multiarray and ufunc, causing circular dependencies. I don't think
> they
> > were there before, and it makes building numpy with a
> dependency-based
> > tool like scons or waf extremely difficult.
> >
> >
> > This particular separation of two components felt somewhat artificial to
> > me.  For the iterator buffering, as an example, I initially thought it
> > would be good to use the same default as the ufuncs do.  Instead I ended
> > up just using a hard-coded default.
>
> Not sure I understand the exact argument, but if it is just a matter of
> getting default values, it is easy to share them using a common header.
>

I guess my argument was just that a few more ufunc features need to migrate
to the core.  I don't think this needs to be done at the moment, though.


>
> > I think the error handling policy
> > in the ufuncs could also be useful for computations in the core.
>
> At the moment, ufunc and multiarray are separate extensions. If each
> depends on the other API-wise, it is an issue. If there are some
> commonalities, then they can be put in a separate extensions. They are
> already way too big as they are (historical reasons), we should really
> fight this for maintanability.
>
> I realize the code organization and the build stuff is a bit of a mess
> at the moment, so if you have questions on that aspect, feel free to ask
> clarification or help,


Yeah, I understand it's the result of organic growth and merging from many
different sources. The core library should probably become layered in a
manner roughly as follows, with each layer depending only on the previous
APIs.  This is what Travis was getting at, I believe, with the generator
array idea affecting mainly the Storage and Iteration APIs, generalizing
them so that new storage and iteration methods can be plugged in.

Data Type API: data type numbers, casting, byte-swapping, etc.
Array Storage API: array creation/manipulation/deletion.
Array Iteration API: array iterators, copying routines.
Array Calculation API: typedefs for various types of calculation
functions, common calculation idioms, ufunc creation API, etc.

Then, the ufunc module would use the Array Calculation API to implement all
the ufuncs and other routines like inner, dot, trace, diag, tensordot,
einsum, etc.

-Mark
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Please keep Numpy master working on Py3

[David]

On 02/02/2011 02:57 PM, Mark Wiebe wrote:
> On Tue, Feb 1, 2011 at 9:49 PM, David  > wrote:
>
> 
>
>
>  > In the meantime, I put the relevant header in
> numpy/core/src/private, to
>  > make the dependency clearer.
>
> Following that argument, there are other unwanted dependencies between
> multiarray and ufunc, causing circular dependencies. I don't think they
> were there before, and it makes building numpy with a dependency-based
> tool like scons or waf extremely difficult.
>
>
> This particular separation of two components felt somewhat artificial to
> me.  For the iterator buffering, as an example, I initially thought it
> would be good to use the same default as the ufuncs do.  Instead I ended
> up just using a hard-coded default.

Not sure I understand the exact argument, but if it is just a matter of 
getting default values, it is easy to share them using a common header.

> I think the error handling policy
> in the ufuncs could also be useful for computations in the core.

At the moment, ufunc and multiarray are separate extensions. If each 
depends on the other API-wise, it is an issue. If there are some 
commonalities, then they can be put in a separate extensions. They are 
already way too big as they are (historical reasons), we should really 
fight this for maintanability.

I realize the code organization and the build stuff is a bit of a mess 
at the moment, so if you have questions on that aspect, feel free to ask 
clarification or help,

cheers,

David
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Please keep Numpy master working on Py3

[Mark Wiebe]

On Tue, Feb 1, 2011 at 9:49 PM, David  wrote:

> 
>

> In the meantime, I put the relevant header in numpy/core/src/private, to
> > make the dependency clearer.
>
> Following that argument, there are other unwanted dependencies between
> multiarray and ufunc, causing circular dependencies. I don't think they
> were there before, and it makes building numpy with a dependency-based
> tool like scons or waf extremely difficult.
>
>
This particular separation of two components felt somewhat artificial to me.
 For the iterator buffering, as an example, I initially thought it would be
good to use the same default as the ufuncs do.  Instead I ended up just
using a hard-coded default.  I think the error handling policy in the ufuncs
could also be useful for computations in the core.  This can't be changed
while maintaining ABI compatibility, though.

-Mark
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Please keep Numpy master working on Py3

[David]

On 02/02/2011 01:53 PM, David wrote:
> On 02/02/2011 08:58 AM, Pauli Virtanen wrote:
>> Hi,
>>
>> The master branch did not build today on Python 3. Please make sure that
>> your code works correctly also on Python 3, before pushing it.
>>
>>   ***
>>
>> I mostly fixed the stuff for now, mostly just the usual bytes vs unicode.
>>
>> On Python 3, the tests however give two non-obvious failures -- I'm not
>> sure if it's just a matter of the raised exception having a different
>> type on Py2 vs Py3, or if it reflects something going wrong. Mark, do you
>> have ideas?
>
> Following the merge on Mark's code, I am stlightly concerned about the
> dependency between ufunc and multiarray (i.e. ufunc include header in
> multiarray).
>
> In the meantime, I put the relevant header in numpy/core/src/private, to
> make the dependency clearer.

Following that argument, there are other unwanted dependencies between 
multiarray and ufunc, causing circular dependencies. I don't think they 
were there before, and it makes building numpy with a dependency-based 
tool like scons or waf extremely difficult.

cheers,

David
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Please keep Numpy master working on Py3

[David]

On 02/02/2011 08:58 AM, Pauli Virtanen wrote:
> Hi,
>
> The master branch did not build today on Python 3. Please make sure that
> your code works correctly also on Python 3, before pushing it.
>
>  ***
>
> I mostly fixed the stuff for now, mostly just the usual bytes vs unicode.
>
> On Python 3, the tests however give two non-obvious failures -- I'm not
> sure if it's just a matter of the raised exception having a different
> type on Py2 vs Py3, or if it reflects something going wrong. Mark, do you
> have ideas?

Following the merge on Mark's code, I am stlightly concerned about the 
dependency between ufunc and multiarray (i.e. ufunc include header in 
multiarray).

In the meantime, I put the relevant header in numpy/core/src/private, to 
make the dependency clearer.

cheers,

David
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Please keep Numpy master working on Py3

[Pauli Virtanen]

On Tue, 01 Feb 2011 16:42:04 -0800, Mark Wiebe wrote:
[clip]
> Sorry about that, I had hoped to be able to use the build-bot to
> test/diagnose the issues, but it's down and I didn't pursue it further.

Hmm, I see Github's SVN gateway is broken again. Perhaps it could be 
possible to make it work directly against Git... Personally, I would like 
to keep the master branch at a known-good-no-experimental-code state at 
all times; different Python versions one can install locally without too 
much trouble, but for cross-architecture testing of experimental code 
using the buildbot would be very helpful. If we are going to fix up a 
"successor" to the current buildbot at some point, this use case probably 
should be kept in mind.

Pauli

___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Please keep Numpy master working on Py3

[Pauli Virtanen]

On Tue, 01 Feb 2011 16:42:04 -0800, Mark Wiebe wrote:
[clip: problem 1]
> This looks like it's a problem in new_iterator_pywrap.c, in
> PySequenceMethods npyiter_as_sequence.  I got this error when trying to
> slice before implementing the slot sq_slice.  I copied the definition
> from array_as_sequence, which I thought would already have any
> adjustments for Py3.  The value in that slot shouldn't be NULL, though,
> so I don't know why it's giving the error, maybe the struct definition
> changed, and somehow in the array sequence methods that gets masked?  Or
> maybe the assignment slice slot needs to be filled as well, but that
> doesn't seem right to me.

The slot sq_slice is in the struct on Python 3, but the entry is a dummy 
no-op.

It appears slices are supposed to be handled in PyMappingMethod. (If you 
walk through the __getitem__ code for ndarray, you'll note that if it 
encounters a single slice, it falls back to calling the PySequence 
methods :O)

The Python C-API docs don't seem to say much on the status of sq_slice on 
Python 2, but at least on the Python side I know that the corresponding 
__*slice__ hooks are deprecated in favor of the __*item__ routines since 
Python 2.6.

[clip: problem 2]
> I see, in Py3 oct and hex call array_index instead of array_oct and
> array_hex.  

This is because the __oct__ and __hex__ C-slots were removed from 
PyNumberMethods on Py3.

> I think changing the self-reference exceptions in
> int/long/float/oct/hex to TypeError instead of ValueError is probably 
> a good way to make this consistent.

Probably a good idea, since the other errors there are also TypeErrors.

Pauli

___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Please keep Numpy master working on Py3

[Mark Wiebe]

On Tue, Feb 1, 2011 at 3:58 PM, Pauli Virtanen  wrote:

> Hi,
>
> The master branch did not build today on Python 3. Please make sure that
> your code works correctly also on Python 3, before pushing it.
>

Sorry about that, I had hoped to be able to use the build-bot to
test/diagnose the issues, but it's down and I didn't pursue it further.


>
>***
>
> I mostly fixed the stuff for now, mostly just the usual bytes vs unicode.
>
> Thanks.


> On Python 3, the tests however give two non-obvious failures -- I'm not
> sure if it's just a matter of the raised exception having a different
> type on Py2 vs Py3, or if it reflects something going wrong. Mark, do you
> have ideas?
>

 Sure, taking a look.

==
> ERROR: test_new_iterator.test_iter_buffering_delayed_alloc
> --
> Traceback (most recent call last):
>  File "/usr/local/lib/python3.1/dist-packages/nose-1.0.0.dev-py3.1.egg/
> nose/case.py", line 188, in runTest
>self.test(*self.arg)
>  File "/home/pauli/prj/scipy/numpy2/dist/linux/lib/python3.1/site-
> packages/numpy/core/tests/test_new_iterator.py", line 1471, in
> test_iter_buffering_delayed_alloc
>assert_raises(ValueError, lambda i:i[0:2], i)
>  File "/home/pauli/prj/scipy/numpy2/dist/linux/lib/python3.1/site-
> packages/numpy/testing/utils.py", line 983, in assert_raises
>return nose.tools.assert_raises(*args,**kwargs)
>  File "/usr/lib/python3.1/unittest.py", line 589, in assertRaises
>callableObj(*args, **kwargs)
>  File "/home/pauli/prj/scipy/numpy2/dist/linux/lib/python3.1/site-
> packages/numpy/core/tests/test_new_iterator.py", line 1471, in 
>assert_raises(ValueError, lambda i:i[0:2], i)
> TypeError: sequence index must be integer, not 'slice'
>

This looks like it's a problem in new_iterator_pywrap.c, in
PySequenceMethods npyiter_as_sequence.  I got this error when trying to
slice before implementing the slot sq_slice.  I copied the definition from
array_as_sequence, which I thought would already have any adjustments for
Py3.  The value in that slot shouldn't be NULL, though, so I don't know why
it's giving the error, maybe the struct definition changed, and somehow in
the array sequence methods that gets masked?  Or maybe the assignment slice
slot needs to be filled as well, but that doesn't seem right to me.


>
> ==
> ERROR: test_object_array_self_reference (test_regression.TestRegression)
> --
> Traceback (most recent call last):
>  File "/home/pauli/prj/scipy/numpy2/dist/linux/lib/python3.1/site-
> packages/numpy/core/tests/test_regression.py", line 1499, in
> test_object_array_self_reference
>assert_raises(ValueError, oct, a)
>  File "/home/pauli/prj/scipy/numpy2/dist/linux/lib/python3.1/site-
> packages/numpy/testing/utils.py", line 983, in assert_raises
>return nose.tools.assert_raises(*args,**kwargs)
>  File "/usr/lib/python3.1/unittest.py", line 589, in assertRaises
>callableObj(*args, **kwargs)
> TypeError: only integer arrays with one element can be converted to an
> index
>

I see, in Py3 oct and hex call array_index instead of array_oct and
array_hex.  I think changing the self-reference exceptions in
int/long/float/oct/hex to TypeError instead of ValueError is probably a good
way to make this consistent.

-Mark
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion