Re: [Numpy-discussion] Fixing issue of future opaqueness of ndarray this summer
On 14 May 2012 21:36, Dag Sverre Seljebotn d.s.seljeb...@astro.uio.no wrote:
On 05/14/2012 06:31 PM, mark florisson wrote:
On 12 May 2012 22:55, Dag Sverre Seljebotnd.s.seljeb...@astro.uio.no
wrote:
On 05/11/2012 03:37 PM, mark florisson wrote:
On 11 May 2012 12:13, Dag Sverre Seljebotnd.s.seljeb...@astro.uio.no
wrote:
(NumPy devs: I know, I get too many ideas. But this time I *really*
believe
in it, I think this is going to be *huge*. And if Mark F. likes it it's
not
going to be without manpower; and as his mentor I'd pitch in too here
and
there.)
(Mark F.: I believe this is *very* relevant to your GSoC. I certainly
don't
want to micro-manage your GSoC, just have your take.)
Travis, thank you very much for those good words in the NA-mask
interactions... thread. It put most of my concerns away. If anybody is
leaning towards for opaqueness because of its OOP purity, I want to
refer
to
C++ and its walled-garden of ideological purity -- it has, what, 3-4
different OOP array libraries, neither of which is able to out-compete
the
other. Meanwhile the rest of the world happily cooperates using
pointers,
strides, CSR and CSC.
Now, there are limits to what you can do with strides and pointers.
Noone's
denying the need for more. In my mind that's an API where you can do
fetch_block and put_block of cache-sized, N-dimensional blocks on an
array;
but it might be something slightly different.
Here's what I'm asking: DO NOT simply keep extending ndarray and the
NumPy C
API to deal with this issue.
What we need is duck-typing/polymorphism at the C level. If you keep
extending ndarray and the NumPy C API, what we'll have is a one-to-many
relationship: One provider of array technology, multiple consumers
(with
hooks, I'm sure, but all implementations of the hook concept in the
NumPy
world I've seen so far are a total disaster!).
What I think we need instead is something like PEP 3118 for the
abstract
array that is only available block-wise with getters and setters. On
the
Cython list we've decided that what we want for CEP 1000 (for boxing
callbacks etc.) is to extend PyTypeObject with our own fields; we could
create CEP 1001 to solve this issue and make any Python object an
exporter
of block-getter/setter-arrays (better name needed).
What would be exported is (of course) a simple vtable:
typedef struct {
int (*get_block)(void *ctx, ssize_t *upper_left, ssize_t
*lower_right,
...);
...
} block_getter_setter_array_vtable;
Let's please discuss the details *after* the fundamentals. But the
reason
I
put void* there instead of PyObject* is that I hope this could be used
beyond the Python world (say, Python-Julia); the void* would be
handed
to
you at the time you receive the vtable (however we handle that).
I suppose it would also be useful to have some way of predicting the
output format polymorphically for the caller. E.g. dense *
block_diagonal results in block diagonal, but dense + block_diagonal
results in dense, etc. It might be useful for the caller to know
whether it needs to allocate a sparse, dense or block-structured
array. Or maybe the polymorphic function could even do the allocation.
This needs to happen recursively of course, to avoid intermediate
temporaries. The compiler could easily handle that, and so could numpy
when it gets lazy evaluation.
Ah. But that depends too on the computation to be performed too; a)
elementwise, b) axis-wise reductions, c) linear algebra...
In my oomatrix code (please don't look at it, it's shameful) I do this
using
multiple dispatch.
I'd rather ignore this for as long as we can, only implementing a[:] =
...
-- I can't see how decisions here would trickle down to the API that's
used
in the kernel, it's more like a pre-phase, and better treated
orthogonally.
I think if the heavy lifting of allocating output arrays and exporting
these arrays work in numpy, then support in Cython could use that (I
can already hear certain people object to more complicated array stuff
in Cython :). Even better here would be an external project that each
our projects could use (I still think the nditer sorting functionality
of arrays should be numpy-agnostic and externally available).
I agree with the separate project idea. It's trivial for NumPy to
incorporate that as one of its methods for exporting arrays, and I don't
think it makes sense to either build it into Cython, or outright depend
on
NumPy.
Here's what I'd like (working title: NumBridge?).
- Mission: Be the double* + shape + strides in a world where that is
no
longer enough, by providing tight, focused APIs/ABIs that are usable
across
C/Fortran/Python.
I basically want something I can quickly acquire from a NumPy array, then
pass it into my C code without dragging along all the cruft that I don't
need.
- Written in pure C + specs, usable without Python
- PEP 3118 done right, basically
Re: [Numpy-discussion] Fixing issue of future opaqueness of ndarray this summer
On 14 May 2012 21:54, David Cournapeau courn...@gmail.com wrote:
On Mon, May 14, 2012 at 5:31 PM, mark florisson markflorisso...@gmail.com
wrote:
On 12 May 2012 22:55, Dag Sverre Seljebotn d.s.seljeb...@astro.uio.no
wrote:
On 05/11/2012 03:37 PM, mark florisson wrote:
On 11 May 2012 12:13, Dag Sverre Seljebotnd.s.seljeb...@astro.uio.no
wrote:
(NumPy devs: I know, I get too many ideas. But this time I *really*
believe
in it, I think this is going to be *huge*. And if Mark F. likes it
it's
not
going to be without manpower; and as his mentor I'd pitch in too here
and
there.)
(Mark F.: I believe this is *very* relevant to your GSoC. I certainly
don't
want to micro-manage your GSoC, just have your take.)
Travis, thank you very much for those good words in the NA-mask
interactions... thread. It put most of my concerns away. If anybody
is
leaning towards for opaqueness because of its OOP purity, I want to
refer
to
C++ and its walled-garden of ideological purity -- it has, what, 3-4
different OOP array libraries, neither of which is able to out-compete
the
other. Meanwhile the rest of the world happily cooperates using
pointers,
strides, CSR and CSC.
Now, there are limits to what you can do with strides and pointers.
Noone's
denying the need for more. In my mind that's an API where you can do
fetch_block and put_block of cache-sized, N-dimensional blocks on an
array;
but it might be something slightly different.
Here's what I'm asking: DO NOT simply keep extending ndarray and the
NumPy C
API to deal with this issue.
What we need is duck-typing/polymorphism at the C level. If you keep
extending ndarray and the NumPy C API, what we'll have is a
one-to-many
relationship: One provider of array technology, multiple consumers
(with
hooks, I'm sure, but all implementations of the hook concept in the
NumPy
world I've seen so far are a total disaster!).
What I think we need instead is something like PEP 3118 for the
abstract
array that is only available block-wise with getters and setters. On
the
Cython list we've decided that what we want for CEP 1000 (for boxing
callbacks etc.) is to extend PyTypeObject with our own fields; we
could
create CEP 1001 to solve this issue and make any Python object an
exporter
of block-getter/setter-arrays (better name needed).
What would be exported is (of course) a simple vtable:
typedef struct {
int (*get_block)(void *ctx, ssize_t *upper_left, ssize_t
*lower_right,
...);
...
} block_getter_setter_array_vtable;
Let's please discuss the details *after* the fundamentals. But the
reason
I
put void* there instead of PyObject* is that I hope this could be used
beyond the Python world (say, Python-Julia); the void* would be
handed
to
you at the time you receive the vtable (however we handle that).
I suppose it would also be useful to have some way of predicting the
output format polymorphically for the caller. E.g. dense *
block_diagonal results in block diagonal, but dense + block_diagonal
results in dense, etc. It might be useful for the caller to know
whether it needs to allocate a sparse, dense or block-structured
array. Or maybe the polymorphic function could even do the allocation.
This needs to happen recursively of course, to avoid intermediate
temporaries. The compiler could easily handle that, and so could numpy
when it gets lazy evaluation.
Ah. But that depends too on the computation to be performed too; a)
elementwise, b) axis-wise reductions, c) linear algebra...
In my oomatrix code (please don't look at it, it's shameful) I do this
using
multiple dispatch.
I'd rather ignore this for as long as we can, only implementing a[:] =
...
-- I can't see how decisions here would trickle down to the API that's
used
in the kernel, it's more like a pre-phase, and better treated
orthogonally.
I think if the heavy lifting of allocating output arrays and exporting
these arrays work in numpy, then support in Cython could use that (I
can already hear certain people object to more complicated array stuff
in Cython :). Even better here would be an external project that each
our projects could use (I still think the nditer sorting functionality
of arrays should be numpy-agnostic and externally available).
I agree with the separate project idea. It's trivial for NumPy to
incorporate that as one of its methods for exporting arrays, and I don't
think it makes sense to either build it into Cython, or outright depend
on
NumPy.
Here's what I'd like (working title: NumBridge?).
- Mission: Be the double* + shape + strides in a world where that is
no
longer enough, by providing tight, focused APIs/ABIs that are usable
across
C/Fortran/Python.
I basically want something I can quickly acquire from a NumPy array,
then
pass it into my C
Re: [Numpy-discussion] Fixing issue of future opaqueness of ndarray this summer
On 05/15/2012 12:42 PM, mark florisson wrote:
On 14 May 2012 21:36, Dag Sverre Seljebotnd.s.seljeb...@astro.uio.no wrote:
On 05/14/2012 06:31 PM, mark florisson wrote:
On 12 May 2012 22:55, Dag Sverre Seljebotnd.s.seljeb...@astro.uio.no
wrote:
On 05/11/2012 03:37 PM, mark florisson wrote:
On 11 May 2012 12:13, Dag Sverre Seljebotnd.s.seljeb...@astro.uio.no
wrote:
(NumPy devs: I know, I get too many ideas. But this time I *really*
believe
in it, I think this is going to be *huge*. And if Mark F. likes it it's
not
going to be without manpower; and as his mentor I'd pitch in too here
and
there.)
(Mark F.: I believe this is *very* relevant to your GSoC. I certainly
don't
want to micro-manage your GSoC, just have your take.)
Travis, thank you very much for those good words in the NA-mask
interactions... thread. It put most of my concerns away. If anybody is
leaning towards for opaqueness because of its OOP purity, I want to
refer
to
C++ and its walled-garden of ideological purity -- it has, what, 3-4
different OOP array libraries, neither of which is able to out-compete
the
other. Meanwhile the rest of the world happily cooperates using
pointers,
strides, CSR and CSC.
Now, there are limits to what you can do with strides and pointers.
Noone's
denying the need for more. In my mind that's an API where you can do
fetch_block and put_block of cache-sized, N-dimensional blocks on an
array;
but it might be something slightly different.
Here's what I'm asking: DO NOT simply keep extending ndarray and the
NumPy C
API to deal with this issue.
What we need is duck-typing/polymorphism at the C level. If you keep
extending ndarray and the NumPy C API, what we'll have is a one-to-many
relationship: One provider of array technology, multiple consumers
(with
hooks, I'm sure, but all implementations of the hook concept in the
NumPy
world I've seen so far are a total disaster!).
What I think we need instead is something like PEP 3118 for the
abstract
array that is only available block-wise with getters and setters. On
the
Cython list we've decided that what we want for CEP 1000 (for boxing
callbacks etc.) is to extend PyTypeObject with our own fields; we could
create CEP 1001 to solve this issue and make any Python object an
exporter
of block-getter/setter-arrays (better name needed).
What would be exported is (of course) a simple vtable:
typedef struct {
int (*get_block)(void *ctx, ssize_t *upper_left, ssize_t
*lower_right,
...);
...
} block_getter_setter_array_vtable;
Let's please discuss the details *after* the fundamentals. But the
reason
I
put void* there instead of PyObject* is that I hope this could be used
beyond the Python world (say, Python-Julia); the void* would be
handed
to
you at the time you receive the vtable (however we handle that).
I suppose it would also be useful to have some way of predicting the
output format polymorphically for the caller. E.g. dense *
block_diagonal results in block diagonal, but dense + block_diagonal
results in dense, etc. It might be useful for the caller to know
whether it needs to allocate a sparse, dense or block-structured
array. Or maybe the polymorphic function could even do the allocation.
This needs to happen recursively of course, to avoid intermediate
temporaries. The compiler could easily handle that, and so could numpy
when it gets lazy evaluation.
Ah. But that depends too on the computation to be performed too; a)
elementwise, b) axis-wise reductions, c) linear algebra...
In my oomatrix code (please don't look at it, it's shameful) I do this
using
multiple dispatch.
I'd rather ignore this for as long as we can, only implementing a[:] =
...
-- I can't see how decisions here would trickle down to the API that's
used
in the kernel, it's more like a pre-phase, and better treated
orthogonally.
I think if the heavy lifting of allocating output arrays and exporting
these arrays work in numpy, then support in Cython could use that (I
can already hear certain people object to more complicated array stuff
in Cython :). Even better here would be an external project that each
our projects could use (I still think the nditer sorting functionality
of arrays should be numpy-agnostic and externally available).
I agree with the separate project idea. It's trivial for NumPy to
incorporate that as one of its methods for exporting arrays, and I don't
think it makes sense to either build it into Cython, or outright depend
on
NumPy.
Here's what I'd like (working title: NumBridge?).
- Mission: Be the double* + shape + strides in a world where that is
no
longer enough, by providing tight, focused APIs/ABIs that are usable
across
C/Fortran/Python.
I basically want something I can quickly acquire from a NumPy array, then
pass it into my C code without dragging along all the cruft that I don't
need.
- Written in pure C + specs, usable
Re: [Numpy-discussion] Fixing issue of future opaqueness of ndarray this summer
Hi, On Tue, May 15, 2012 at 6:42 AM, mark florisson markflorisso...@gmail.com wrote: I assumed as much, in any case I was going to start with dense arrays, simply because they are in common use and well-defined at this point. Maybe what we really want is just lazy evaluation that works for different array layouts and storage mechanisms, and a smart JIT that can evaluate our linear algebra and array expressions in an optimal fashion (memory/cache-wise, but also out-of-core wise). This probably means writing everything in a high level language, because even if your Fortran routines themselves would use your lazy evaluation library, your linear algebra library wouldn't for sure :) Anyway, I'm going to be pragmatic and investigate how much of Theano we can reuse in Cython now. Don't hesitate to contact me or the mailing lists for questions/comments/anything. We can also arrange skype meeting if this help. Fred ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Fixing issue of future opaqueness of ndarray this summer
On 05/13/2012 12:27 AM, Charles R Harris wrote:
On Sat, May 12, 2012 at 3:55 PM, Dag Sverre Seljebotn
d.s.seljeb...@astro.uio.no mailto:d.s.seljeb...@astro.uio.no wrote:
On 05/11/2012 03:37 PM, mark florisson wrote:
On 11 May 2012 12:13, Dag Sverre
Seljebotnd.s.seljeb...@astro.uio.no
mailto:d.s.seljeb...@astro.uio.no wrote:
(NumPy devs: I know, I get too many ideas. But this time I
*really* believe
in it, I think this is going to be *huge*. And if Mark F. likes
it it's not
going to be without manpower; and as his mentor I'd pitch in too
here and
there.)
(Mark F.: I believe this is *very* relevant to your GSoC. I
certainly don't
want to micro-manage your GSoC, just have your take.)
Travis, thank you very much for those good words in the NA-mask
interactions... thread. It put most of my concerns away. If
anybody is
leaning towards for opaqueness because of its OOP purity, I want
to refer to
C++ and its walled-garden of ideological purity -- it has, what, 3-4
different OOP array libraries, neither of which is able to
out-compete the
other. Meanwhile the rest of the world happily cooperates using
pointers,
strides, CSR and CSC.
Now, there are limits to what you can do with strides and
pointers. Noone's
denying the need for more. In my mind that's an API where you can do
fetch_block and put_block of cache-sized, N-dimensional blocks
on an array;
but it might be something slightly different.
Here's what I'm asking: DO NOT simply keep extending ndarray and
the NumPy C
API to deal with this issue.
What we need is duck-typing/polymorphism at the C level. If you keep
extending ndarray and the NumPy C API, what we'll have is a
one-to-many
relationship: One provider of array technology, multiple
consumers (with
hooks, I'm sure, but all implementations of the hook concept in
the NumPy
world I've seen so far are a total disaster!).
What I think we need instead is something like PEP 3118 for the
abstract
array that is only available block-wise with getters and
setters. On the
Cython list we've decided that what we want for CEP 1000 (for boxing
callbacks etc.) is to extend PyTypeObject with our own fields;
we could
create CEP 1001 to solve this issue and make any Python object
an exporter
of block-getter/setter-arrays (better name needed).
What would be exported is (of course) a simple vtable:
typedef struct {
int (*get_block)(void *ctx, ssize_t *upper_left, ssize_t
*lower_right,
...);
...
} block_getter_setter_array_vtable;
Let's please discuss the details *after* the fundamentals. But
the reason I
put void* there instead of PyObject* is that I hope this could
be used
beyond the Python world (say, Python-Julia); the void* would
be handed to
you at the time you receive the vtable (however we handle that).
I suppose it would also be useful to have some way of predicting the
output format polymorphically for the caller. E.g. dense *
block_diagonal results in block diagonal, but dense + block_diagonal
results in dense, etc. It might be useful for the caller to know
whether it needs to allocate a sparse, dense or block-structured
array. Or maybe the polymorphic function could even do the
allocation.
This needs to happen recursively of course, to avoid intermediate
temporaries. The compiler could easily handle that, and so could
numpy
when it gets lazy evaluation.
Ah. But that depends too on the computation to be performed too; a)
elementwise, b) axis-wise reductions, c) linear algebra...
In my oomatrix code (please don't look at it, it's shameful) I do this
using multiple dispatch.
I'd rather ignore this for as long as we can, only implementing a[:] =
... -- I can't see how decisions here would trickle down to the API
that's used in the kernel, it's more like a pre-phase, and better
treated orthogonally.
I think if the heavy lifting of allocating output arrays and
exporting
these arrays work in numpy, then support in Cython could use that (I
can already hear certain people object to more complicated array
stuff
in Cython :). Even better here would be an external project that each
our projects could use (I still think the nditer sorting
functionality
of arrays should be numpy-agnostic and externally available).
I agree with the separate project idea. It's trivial for NumPy to
incorporate that as one of its methods for exporting arrays, and I don't
think
Re: [Numpy-discussion] Fixing issue of future opaqueness of ndarray this summer
On 12 May 2012 22:55, Dag Sverre Seljebotn d.s.seljeb...@astro.uio.no wrote:
On 05/11/2012 03:37 PM, mark florisson wrote:
On 11 May 2012 12:13, Dag Sverre Seljebotnd.s.seljeb...@astro.uio.no
wrote:
(NumPy devs: I know, I get too many ideas. But this time I *really*
believe
in it, I think this is going to be *huge*. And if Mark F. likes it it's
not
going to be without manpower; and as his mentor I'd pitch in too here and
there.)
(Mark F.: I believe this is *very* relevant to your GSoC. I certainly
don't
want to micro-manage your GSoC, just have your take.)
Travis, thank you very much for those good words in the NA-mask
interactions... thread. It put most of my concerns away. If anybody is
leaning towards for opaqueness because of its OOP purity, I want to refer
to
C++ and its walled-garden of ideological purity -- it has, what, 3-4
different OOP array libraries, neither of which is able to out-compete
the
other. Meanwhile the rest of the world happily cooperates using pointers,
strides, CSR and CSC.
Now, there are limits to what you can do with strides and pointers.
Noone's
denying the need for more. In my mind that's an API where you can do
fetch_block and put_block of cache-sized, N-dimensional blocks on an
array;
but it might be something slightly different.
Here's what I'm asking: DO NOT simply keep extending ndarray and the
NumPy C
API to deal with this issue.
What we need is duck-typing/polymorphism at the C level. If you keep
extending ndarray and the NumPy C API, what we'll have is a one-to-many
relationship: One provider of array technology, multiple consumers (with
hooks, I'm sure, but all implementations of the hook concept in the NumPy
world I've seen so far are a total disaster!).
What I think we need instead is something like PEP 3118 for the
abstract
array that is only available block-wise with getters and setters. On the
Cython list we've decided that what we want for CEP 1000 (for boxing
callbacks etc.) is to extend PyTypeObject with our own fields; we could
create CEP 1001 to solve this issue and make any Python object an
exporter
of block-getter/setter-arrays (better name needed).
What would be exported is (of course) a simple vtable:
typedef struct {
int (*get_block)(void *ctx, ssize_t *upper_left, ssize_t *lower_right,
...);
...
} block_getter_setter_array_vtable;
Let's please discuss the details *after* the fundamentals. But the reason
I
put void* there instead of PyObject* is that I hope this could be used
beyond the Python world (say, Python-Julia); the void* would be handed
to
you at the time you receive the vtable (however we handle that).
I suppose it would also be useful to have some way of predicting the
output format polymorphically for the caller. E.g. dense *
block_diagonal results in block diagonal, but dense + block_diagonal
results in dense, etc. It might be useful for the caller to know
whether it needs to allocate a sparse, dense or block-structured
array. Or maybe the polymorphic function could even do the allocation.
This needs to happen recursively of course, to avoid intermediate
temporaries. The compiler could easily handle that, and so could numpy
when it gets lazy evaluation.
Ah. But that depends too on the computation to be performed too; a)
elementwise, b) axis-wise reductions, c) linear algebra...
In my oomatrix code (please don't look at it, it's shameful) I do this using
multiple dispatch.
I'd rather ignore this for as long as we can, only implementing a[:] = ...
-- I can't see how decisions here would trickle down to the API that's used
in the kernel, it's more like a pre-phase, and better treated orthogonally.
I think if the heavy lifting of allocating output arrays and exporting
these arrays work in numpy, then support in Cython could use that (I
can already hear certain people object to more complicated array stuff
in Cython :). Even better here would be an external project that each
our projects could use (I still think the nditer sorting functionality
of arrays should be numpy-agnostic and externally available).
I agree with the separate project idea. It's trivial for NumPy to
incorporate that as one of its methods for exporting arrays, and I don't
think it makes sense to either build it into Cython, or outright depend on
NumPy.
Here's what I'd like (working title: NumBridge?).
- Mission: Be the double* + shape + strides in a world where that is no
longer enough, by providing tight, focused APIs/ABIs that are usable across
C/Fortran/Python.
I basically want something I can quickly acquire from a NumPy array, then
pass it into my C code without dragging along all the cruft that I don't
need.
- Written in pure C + specs, usable without Python
- PEP 3118 done right, basically semi-standardize the internal Cython
memoryview ABI and get something that's passable on stack
- Get block get/put API
- Iterator APIs
- Utility
Re: [Numpy-discussion] Fixing issue of future opaqueness of ndarray this summer
On 05/14/2012 06:31 PM, mark florisson wrote:
On 12 May 2012 22:55, Dag Sverre Seljebotnd.s.seljeb...@astro.uio.no wrote:
On 05/11/2012 03:37 PM, mark florisson wrote:
On 11 May 2012 12:13, Dag Sverre Seljebotnd.s.seljeb...@astro.uio.no
wrote:
(NumPy devs: I know, I get too many ideas. But this time I *really*
believe
in it, I think this is going to be *huge*. And if Mark F. likes it it's
not
going to be without manpower; and as his mentor I'd pitch in too here and
there.)
(Mark F.: I believe this is *very* relevant to your GSoC. I certainly
don't
want to micro-manage your GSoC, just have your take.)
Travis, thank you very much for those good words in the NA-mask
interactions... thread. It put most of my concerns away. If anybody is
leaning towards for opaqueness because of its OOP purity, I want to refer
to
C++ and its walled-garden of ideological purity -- it has, what, 3-4
different OOP array libraries, neither of which is able to out-compete
the
other. Meanwhile the rest of the world happily cooperates using pointers,
strides, CSR and CSC.
Now, there are limits to what you can do with strides and pointers.
Noone's
denying the need for more. In my mind that's an API where you can do
fetch_block and put_block of cache-sized, N-dimensional blocks on an
array;
but it might be something slightly different.
Here's what I'm asking: DO NOT simply keep extending ndarray and the
NumPy C
API to deal with this issue.
What we need is duck-typing/polymorphism at the C level. If you keep
extending ndarray and the NumPy C API, what we'll have is a one-to-many
relationship: One provider of array technology, multiple consumers (with
hooks, I'm sure, but all implementations of the hook concept in the NumPy
world I've seen so far are a total disaster!).
What I think we need instead is something like PEP 3118 for the
abstract
array that is only available block-wise with getters and setters. On the
Cython list we've decided that what we want for CEP 1000 (for boxing
callbacks etc.) is to extend PyTypeObject with our own fields; we could
create CEP 1001 to solve this issue and make any Python object an
exporter
of block-getter/setter-arrays (better name needed).
What would be exported is (of course) a simple vtable:
typedef struct {
int (*get_block)(void *ctx, ssize_t *upper_left, ssize_t *lower_right,
...);
...
} block_getter_setter_array_vtable;
Let's please discuss the details *after* the fundamentals. But the reason
I
put void* there instead of PyObject* is that I hope this could be used
beyond the Python world (say, Python-Julia); the void* would be handed
to
you at the time you receive the vtable (however we handle that).
I suppose it would also be useful to have some way of predicting the
output format polymorphically for the caller. E.g. dense *
block_diagonal results in block diagonal, but dense + block_diagonal
results in dense, etc. It might be useful for the caller to know
whether it needs to allocate a sparse, dense or block-structured
array. Or maybe the polymorphic function could even do the allocation.
This needs to happen recursively of course, to avoid intermediate
temporaries. The compiler could easily handle that, and so could numpy
when it gets lazy evaluation.
Ah. But that depends too on the computation to be performed too; a)
elementwise, b) axis-wise reductions, c) linear algebra...
In my oomatrix code (please don't look at it, it's shameful) I do this using
multiple dispatch.
I'd rather ignore this for as long as we can, only implementing a[:] = ...
-- I can't see how decisions here would trickle down to the API that's used
in the kernel, it's more like a pre-phase, and better treated orthogonally.
I think if the heavy lifting of allocating output arrays and exporting
these arrays work in numpy, then support in Cython could use that (I
can already hear certain people object to more complicated array stuff
in Cython :). Even better here would be an external project that each
our projects could use (I still think the nditer sorting functionality
of arrays should be numpy-agnostic and externally available).
I agree with the separate project idea. It's trivial for NumPy to
incorporate that as one of its methods for exporting arrays, and I don't
think it makes sense to either build it into Cython, or outright depend on
NumPy.
Here's what I'd like (working title: NumBridge?).
- Mission: Be the double* + shape + strides in a world where that is no
longer enough, by providing tight, focused APIs/ABIs that are usable across
C/Fortran/Python.
I basically want something I can quickly acquire from a NumPy array, then
pass it into my C code without dragging along all the cruft that I don't
need.
- Written in pure C + specs, usable without Python
- PEP 3118 done right, basically semi-standardize the internal Cython
memoryview ABI and get something that's passable on stack
-
Re: [Numpy-discussion] Fixing issue of future opaqueness of ndarray this summer
On 05/14/2012 10:36 PM, Dag Sverre Seljebotn wrote:
On 05/14/2012 06:31 PM, mark florisson wrote:
On 12 May 2012 22:55, Dag Sverre Seljebotnd.s.seljeb...@astro.uio.no
wrote:
On 05/11/2012 03:37 PM, mark florisson wrote:
On 11 May 2012 12:13, Dag Sverre Seljebotnd.s.seljeb...@astro.uio.no
wrote:
(NumPy devs: I know, I get too many ideas. But this time I *really*
believe
in it, I think this is going to be *huge*. And if Mark F. likes it it's
not
going to be without manpower; and as his mentor I'd pitch in too here and
there.)
(Mark F.: I believe this is *very* relevant to your GSoC. I certainly
don't
want to micro-manage your GSoC, just have your take.)
Travis, thank you very much for those good words in the NA-mask
interactions... thread. It put most of my concerns away. If anybody is
leaning towards for opaqueness because of its OOP purity, I want to refer
to
C++ and its walled-garden of ideological purity -- it has, what, 3-4
different OOP array libraries, neither of which is able to out-compete
the
other. Meanwhile the rest of the world happily cooperates using pointers,
strides, CSR and CSC.
Now, there are limits to what you can do with strides and pointers.
Noone's
denying the need for more. In my mind that's an API where you can do
fetch_block and put_block of cache-sized, N-dimensional blocks on an
array;
but it might be something slightly different.
Here's what I'm asking: DO NOT simply keep extending ndarray and the
NumPy C
API to deal with this issue.
What we need is duck-typing/polymorphism at the C level. If you keep
extending ndarray and the NumPy C API, what we'll have is a one-to-many
relationship: One provider of array technology, multiple consumers (with
hooks, I'm sure, but all implementations of the hook concept in the NumPy
world I've seen so far are a total disaster!).
What I think we need instead is something like PEP 3118 for the
abstract
array that is only available block-wise with getters and setters. On the
Cython list we've decided that what we want for CEP 1000 (for boxing
callbacks etc.) is to extend PyTypeObject with our own fields; we could
create CEP 1001 to solve this issue and make any Python object an
exporter
of block-getter/setter-arrays (better name needed).
What would be exported is (of course) a simple vtable:
typedef struct {
int (*get_block)(void *ctx, ssize_t *upper_left, ssize_t
*lower_right,
...);
...
} block_getter_setter_array_vtable;
Let's please discuss the details *after* the fundamentals. But the reason
I
put void* there instead of PyObject* is that I hope this could be used
beyond the Python world (say, Python-Julia); the void* would be handed
to
you at the time you receive the vtable (however we handle that).
I suppose it would also be useful to have some way of predicting the
output format polymorphically for the caller. E.g. dense *
block_diagonal results in block diagonal, but dense + block_diagonal
results in dense, etc. It might be useful for the caller to know
whether it needs to allocate a sparse, dense or block-structured
array. Or maybe the polymorphic function could even do the allocation.
This needs to happen recursively of course, to avoid intermediate
temporaries. The compiler could easily handle that, and so could numpy
when it gets lazy evaluation.
Ah. But that depends too on the computation to be performed too; a)
elementwise, b) axis-wise reductions, c) linear algebra...
In my oomatrix code (please don't look at it, it's shameful) I do this using
multiple dispatch.
I'd rather ignore this for as long as we can, only implementing a[:] = ...
-- I can't see how decisions here would trickle down to the API that's used
in the kernel, it's more like a pre-phase, and better treated orthogonally.
I think if the heavy lifting of allocating output arrays and exporting
these arrays work in numpy, then support in Cython could use that (I
can already hear certain people object to more complicated array stuff
in Cython :). Even better here would be an external project that each
our projects could use (I still think the nditer sorting functionality
of arrays should be numpy-agnostic and externally available).
I agree with the separate project idea. It's trivial for NumPy to
incorporate that as one of its methods for exporting arrays, and I don't
think it makes sense to either build it into Cython, or outright depend on
NumPy.
Here's what I'd like (working title: NumBridge?).
- Mission: Be the double* + shape + strides in a world where that is no
longer enough, by providing tight, focused APIs/ABIs that are usable across
C/Fortran/Python.
I basically want something I can quickly acquire from a NumPy array, then
pass it into my C code without dragging along all the cruft that I don't
need.
- Written in pure C + specs, usable without Python
- PEP 3118 done right, basically semi-standardize the internal Cython
Re: [Numpy-discussion] Fixing issue of future opaqueness of ndarray this summer
On Mon, May 14, 2012 at 5:31 PM, mark florisson
markflorisso...@gmail.comwrote:
On 12 May 2012 22:55, Dag Sverre Seljebotn d.s.seljeb...@astro.uio.no
wrote:
On 05/11/2012 03:37 PM, mark florisson wrote:
On 11 May 2012 12:13, Dag Sverre Seljebotnd.s.seljeb...@astro.uio.no
wrote:
(NumPy devs: I know, I get too many ideas. But this time I *really*
believe
in it, I think this is going to be *huge*. And if Mark F. likes it it's
not
going to be without manpower; and as his mentor I'd pitch in too here
and
there.)
(Mark F.: I believe this is *very* relevant to your GSoC. I certainly
don't
want to micro-manage your GSoC, just have your take.)
Travis, thank you very much for those good words in the NA-mask
interactions... thread. It put most of my concerns away. If anybody is
leaning towards for opaqueness because of its OOP purity, I want to
refer
to
C++ and its walled-garden of ideological purity -- it has, what, 3-4
different OOP array libraries, neither of which is able to out-compete
the
other. Meanwhile the rest of the world happily cooperates using
pointers,
strides, CSR and CSC.
Now, there are limits to what you can do with strides and pointers.
Noone's
denying the need for more. In my mind that's an API where you can do
fetch_block and put_block of cache-sized, N-dimensional blocks on an
array;
but it might be something slightly different.
Here's what I'm asking: DO NOT simply keep extending ndarray and the
NumPy C
API to deal with this issue.
What we need is duck-typing/polymorphism at the C level. If you keep
extending ndarray and the NumPy C API, what we'll have is a one-to-many
relationship: One provider of array technology, multiple consumers
(with
hooks, I'm sure, but all implementations of the hook concept in the
NumPy
world I've seen so far are a total disaster!).
What I think we need instead is something like PEP 3118 for the
abstract
array that is only available block-wise with getters and setters. On
the
Cython list we've decided that what we want for CEP 1000 (for boxing
callbacks etc.) is to extend PyTypeObject with our own fields; we could
create CEP 1001 to solve this issue and make any Python object an
exporter
of block-getter/setter-arrays (better name needed).
What would be exported is (of course) a simple vtable:
typedef struct {
int (*get_block)(void *ctx, ssize_t *upper_left, ssize_t
*lower_right,
...);
...
} block_getter_setter_array_vtable;
Let's please discuss the details *after* the fundamentals. But the
reason
I
put void* there instead of PyObject* is that I hope this could be used
beyond the Python world (say, Python-Julia); the void* would be
handed
to
you at the time you receive the vtable (however we handle that).
I suppose it would also be useful to have some way of predicting the
output format polymorphically for the caller. E.g. dense *
block_diagonal results in block diagonal, but dense + block_diagonal
results in dense, etc. It might be useful for the caller to know
whether it needs to allocate a sparse, dense or block-structured
array. Or maybe the polymorphic function could even do the allocation.
This needs to happen recursively of course, to avoid intermediate
temporaries. The compiler could easily handle that, and so could numpy
when it gets lazy evaluation.
Ah. But that depends too on the computation to be performed too; a)
elementwise, b) axis-wise reductions, c) linear algebra...
In my oomatrix code (please don't look at it, it's shameful) I do this
using
multiple dispatch.
I'd rather ignore this for as long as we can, only implementing a[:] =
...
-- I can't see how decisions here would trickle down to the API that's
used
in the kernel, it's more like a pre-phase, and better treated
orthogonally.
I think if the heavy lifting of allocating output arrays and exporting
these arrays work in numpy, then support in Cython could use that (I
can already hear certain people object to more complicated array stuff
in Cython :). Even better here would be an external project that each
our projects could use (I still think the nditer sorting functionality
of arrays should be numpy-agnostic and externally available).
I agree with the separate project idea. It's trivial for NumPy to
incorporate that as one of its methods for exporting arrays, and I don't
think it makes sense to either build it into Cython, or outright depend
on
NumPy.
Here's what I'd like (working title: NumBridge?).
- Mission: Be the double* + shape + strides in a world where that is
no
longer enough, by providing tight, focused APIs/ABIs that are usable
across
C/Fortran/Python.
I basically want something I can quickly acquire from a NumPy array, then
pass it into my C code without dragging along all the cruft that I don't
need.
- Written in pure C + specs,
Re: [Numpy-discussion] Fixing issue of future opaqueness of ndarray this summer
On 05/11/2012 10:10 PM, Mark Wiebe wrote:
On Fri, May 11, 2012 at 6:13 AM, Dag Sverre Seljebotn
d.s.seljeb...@astro.uio.no mailto:d.s.seljeb...@astro.uio.no wrote:
(NumPy devs: I know, I get too many ideas. But this time I *really*
believe in it, I think this is going to be *huge*. And if Mark F. likes
it it's not going to be without manpower; and as his mentor I'd pitch in
too here and there.)
(Mark F.: I believe this is *very* relevant to your GSoC. I certainly
don't want to micro-manage your GSoC, just have your take.)
Travis, thank you very much for those good words in the NA-mask
interactions... thread. It put most of my concerns away. If anybody is
leaning towards for opaqueness because of its OOP purity, I want to
refer to C++ and its walled-garden of ideological purity -- it has,
what, 3-4 different OOP array libraries, neither of which is able to
out-compete the other. Meanwhile the rest of the world happily
cooperates using pointers, strides, CSR and CSC.
Now, there are limits to what you can do with strides and pointers.
Noone's denying the need for more. In my mind that's an API where you
can do fetch_block and put_block of cache-sized, N-dimensional blocks on
an array; but it might be something slightly different.
Here's what I'm asking: DO NOT simply keep extending ndarray and the
NumPy C API to deal with this issue.
What we need is duck-typing/polymorphism at the C level. If you keep
extending ndarray and the NumPy C API, what we'll have is a one-to-many
relationship: One provider of array technology, multiple consumers (with
hooks, I'm sure, but all implementations of the hook concept in the
NumPy world I've seen so far are a total disaster!).
There is similar intent behind an idea I raised last summer here:
http://mail.scipy.org/pipermail/numpy-discussion/2011-June/056945.html
I stopped doing anything on it when considering the scope of it with the
linear algebra functions included like Pauli suggested. Nathaniel did an
experimental implementation of some parts of the idea in Python here:
Hmm. Ufuncs. I'll think more about that, and read the thread again, to
see if I can discover the connection to my proposal above. (Re your
thread, I always wished for something based on multiple dispatch rather
than a priority battle to resolve NumPy operators; you really only have
a partial ordering between libraries wanting to interact with ndarrays,
not a total ordering)
Re linear algebra, I have, since I use it in my PhD, been working on and
off the past 4 months on a library for polymorphic linear algebra.
There's something analogue to the array and ufunc distinction of NumPy,
but the ufunc part is designed very differently (and based on multiple
dispatch).
Anyway, what I was talking about was basically a new C interface to
arrays that should be more tolerant for implementations that need to be
opaque (like compressed arrays, database connections, my own custom
sparse format...). If that somehow relates with ufuncs, fine with me :-)
https://github.com/njsmith/numpyNEP/blob/master/numpyNEP.py#L107
What I think we need instead is something like PEP 3118 for the
abstract array that is only available block-wise with getters and
setters. On the Cython list we've decided that what we want for CEP 1000
(for boxing callbacks etc.) is to extend PyTypeObject with our own
fields; we could create CEP 1001 to solve this issue and make any Python
object an exporter of block-getter/setter-arrays (better name needed).
What would be exported is (of course) a simple vtable:
typedef struct {
int (*get_block)(void *ctx, ssize_t *upper_left, ssize_t
*lower_right, ...);
...
} block_getter_setter_array_vtable;
Let's please discuss the details *after* the fundamentals. But the
reason I put void* there instead of PyObject* is that I hope this could
be used beyond the Python world (say, Python-Julia); the void* would
be handed to you at the time you receive the vtable (however we handle
that).
I think this would fit neatly in Mark F.'s GSoC (Mark F.?), because you
could embed the block-transposition that's needed for efficient arr +
arr.T at this level.
Imagine being able to do this in Cython:
a[...] = b + c * d
and have that essentially compile to the numexpr blocked approach, *but*
where b, c, and d can have whatever type that exports CEP 1001? So c
could be a diagonal array which uses O(n) storage to export O(n^2)
elements, for instance, and the unrolled Cython code never needs to
know.
As far as NumPy goes, something along these lines should hopefully mean
that new C code being written doesn't rely so much on what exactly goes
into ndarray and what goes into other classes; so that we don't get
Re: [Numpy-discussion] Fixing issue of future opaqueness of ndarray this summer
On 05/11/2012 03:25 PM, mark florisson wrote:
On 11 May 2012 12:13, Dag Sverre Seljebotnd.s.seljeb...@astro.uio.no wrote:
(NumPy devs: I know, I get too many ideas. But this time I *really* believe
in it, I think this is going to be *huge*. And if Mark F. likes it it's not
going to be without manpower; and as his mentor I'd pitch in too here and
there.)
(Mark F.: I believe this is *very* relevant to your GSoC. I certainly don't
want to micro-manage your GSoC, just have your take.)
Travis, thank you very much for those good words in the NA-mask
interactions... thread. It put most of my concerns away. If anybody is
leaning towards for opaqueness because of its OOP purity, I want to refer to
C++ and its walled-garden of ideological purity -- it has, what, 3-4
different OOP array libraries, neither of which is able to out-compete the
other. Meanwhile the rest of the world happily cooperates using pointers,
strides, CSR and CSC.
BTW, this seems like pure hyperbole out of context...the backstory is
that I've talked to people who do sparse linear algebra in C++ using
Boost, iterators, etc. rather than something that could conceivably be
exported with a binary interface; primarily out of a wish to be more
elegant than 'legacy' programming languages that have to resort to
things like CSR. Sure, that's elegant modern C++, but at what cost?
Now, there are limits to what you can do with strides and pointers. Noone's
denying the need for more. In my mind that's an API where you can do
fetch_block and put_block of cache-sized, N-dimensional blocks on an array;
but it might be something slightly different.
Here's what I'm asking: DO NOT simply keep extending ndarray and the NumPy C
API to deal with this issue.
What we need is duck-typing/polymorphism at the C level. If you keep
extending ndarray and the NumPy C API, what we'll have is a one-to-many
relationship: One provider of array technology, multiple consumers (with
hooks, I'm sure, but all implementations of the hook concept in the NumPy
world I've seen so far are a total disaster!).
What I think we need instead is something like PEP 3118 for the abstract
array that is only available block-wise with getters and setters. On the
Cython list we've decided that what we want for CEP 1000 (for boxing
callbacks etc.) is to extend PyTypeObject with our own fields; we could
create CEP 1001 to solve this issue and make any Python object an exporter
of block-getter/setter-arrays (better name needed).
What would be exported is (of course) a simple vtable:
typedef struct {
int (*get_block)(void *ctx, ssize_t *upper_left, ssize_t *lower_right,
...);
...
} block_getter_setter_array_vtable;
Interesting idea, I think I like it. So I suppose these blocks could
even be used in a sparse context, where it returns one-sized blocks
for un-surrounded elements. This means returned blocks can always be
variably sized, correct? So will each block be associated with an
index space? Otherwise, how do we know the element-wise
correspondence? Will there be a default element for missing
(unreturned) values?
Am I correct in assuming elements in returned blocks always return
elements in element-wise order? In case you manually have to block
other parts of an array (lets say it's stored in Fortran order), then
the variable sized block nature may be complicating the tiling
pattern.
- My hunch was that you'd always get something C-contiguous and the
exporter would copy/transpose. But perhaps one wouldn't loose much by
making the block strided and have the consumer do a copy; isn't that
mostly about which side of the boundary to call the copying utility?
- I think iterators over blocks are necesarry as well
- Not sure about variable-sized blocks. The array could already be
stored in cache-sized blocks, and the numerical loops could be optimized
for some block sizes, so there must certainly be some handshaking
mechanism; hopefully it can be made rather elegant. See next point for
one ingredient.
- What must be established is whether a) one should always copy, b)
one can copy if one wants to at a negligible cost, or c) computational
core reaching into the buffer of an exporter is better. This would
trickle down into lots of other decisions I feel (like if you always go
for a rectangular block, or whether getting entire rows like with the
NumPy iteration API is sometimes more valuable).
In addition to the Fortran compilers, one thing to study is
GotoBLAS2/OpenBLAS, which has code for getting a high-quality memory
region, since that apparently made an impact. Not sure how much Intel
platforms though; other platforms have slower TLBs, in fact I think on
non-Intel platforms, TLB is what affected block size, not the cache size.
Let's please discuss the details *after* the fundamentals. But the reason I
put void* there instead of PyObject* is that I hope this could be used
beyond the Python world (say,
Re: [Numpy-discussion] Fixing issue of future opaqueness of ndarray this summer
On 05/12/2012 11:35 PM, Dag Sverre Seljebotn wrote:
On 05/11/2012 03:25 PM, mark florisson wrote:
On 11 May 2012 12:13, Dag Sverre Seljebotnd.s.seljeb...@astro.uio.no
wrote:
(NumPy devs: I know, I get too many ideas. But this time I *really* believe
in it, I think this is going to be *huge*. And if Mark F. likes it it's not
going to be without manpower; and as his mentor I'd pitch in too here and
there.)
(Mark F.: I believe this is *very* relevant to your GSoC. I certainly don't
want to micro-manage your GSoC, just have your take.)
Travis, thank you very much for those good words in the NA-mask
interactions... thread. It put most of my concerns away. If anybody is
leaning towards for opaqueness because of its OOP purity, I want to refer to
C++ and its walled-garden of ideological purity -- it has, what, 3-4
different OOP array libraries, neither of which is able to out-compete the
other. Meanwhile the rest of the world happily cooperates using pointers,
strides, CSR and CSC.
BTW, this seems like pure hyperbole out of context...the backstory is
that I've talked to people who do sparse linear algebra in C++ using
Boost, iterators, etc. rather than something that could conceivably be
exported with a binary interface; primarily out of a wish to be more
elegant than 'legacy' programming languages that have to resort to
things like CSR. Sure, that's elegant modern C++, but at what cost?
Now, there are limits to what you can do with strides and pointers. Noone's
denying the need for more. In my mind that's an API where you can do
fetch_block and put_block of cache-sized, N-dimensional blocks on an array;
but it might be something slightly different.
Here's what I'm asking: DO NOT simply keep extending ndarray and the NumPy C
API to deal with this issue.
What we need is duck-typing/polymorphism at the C level. If you keep
extending ndarray and the NumPy C API, what we'll have is a one-to-many
relationship: One provider of array technology, multiple consumers (with
hooks, I'm sure, but all implementations of the hook concept in the NumPy
world I've seen so far are a total disaster!).
What I think we need instead is something like PEP 3118 for the abstract
array that is only available block-wise with getters and setters. On the
Cython list we've decided that what we want for CEP 1000 (for boxing
callbacks etc.) is to extend PyTypeObject with our own fields; we could
create CEP 1001 to solve this issue and make any Python object an exporter
of block-getter/setter-arrays (better name needed).
What would be exported is (of course) a simple vtable:
typedef struct {
int (*get_block)(void *ctx, ssize_t *upper_left, ssize_t *lower_right,
...);
...
} block_getter_setter_array_vtable;
Interesting idea, I think I like it. So I suppose these blocks could
even be used in a sparse context, where it returns one-sized blocks
for un-surrounded elements. This means returned blocks can always be
variably sized, correct? So will each block be associated with an
index space? Otherwise, how do we know the element-wise
correspondence? Will there be a default element for missing
(unreturned) values?
Am I correct in assuming elements in returned blocks always return
elements in element-wise order? In case you manually have to block
other parts of an array (lets say it's stored in Fortran order), then
the variable sized block nature may be complicating the tiling
pattern.
- My hunch was that you'd always get something C-contiguous and the
exporter would copy/transpose. But perhaps one wouldn't loose much by
making the block strided and have the consumer do a copy; isn't that
mostly about which side of the boundary to call the copying utility?
- I think iterators over blocks are necesarry as well
- Not sure about variable-sized blocks. The array could already be
stored in cache-sized blocks, and the numerical loops could be optimized
for some block sizes, so there must certainly be some handshaking
mechanism; hopefully it can be made rather elegant. See next point for
one ingredient.
- What must be established is whether a) one should always copy, b)
one can copy if one wants to at a negligible cost, or c) computational
core reaching into the buffer of an exporter is better. This would
trickle down into lots of other decisions I feel (like if you always go
for a rectangular block, or whether getting entire rows like with the
NumPy iteration API is sometimes more valuable).
In addition to the Fortran compilers, one thing to study is
GotoBLAS2/OpenBLAS, which has code for getting a high-quality memory
region, since that apparently made an impact. Not sure how much Intel
platforms though; other platforms have slower TLBs, in fact I think on
non-Intel platforms, TLB is what affected block size, not the cache size.
Let's please discuss the details *after* the fundamentals. But the reason I
put void* there instead of
Re: [Numpy-discussion] Fixing issue of future opaqueness of ndarray this summer
On 05/11/2012 03:37 PM, mark florisson wrote:
On 11 May 2012 12:13, Dag Sverre Seljebotnd.s.seljeb...@astro.uio.no wrote:
(NumPy devs: I know, I get too many ideas. But this time I *really* believe
in it, I think this is going to be *huge*. And if Mark F. likes it it's not
going to be without manpower; and as his mentor I'd pitch in too here and
there.)
(Mark F.: I believe this is *very* relevant to your GSoC. I certainly don't
want to micro-manage your GSoC, just have your take.)
Travis, thank you very much for those good words in the NA-mask
interactions... thread. It put most of my concerns away. If anybody is
leaning towards for opaqueness because of its OOP purity, I want to refer to
C++ and its walled-garden of ideological purity -- it has, what, 3-4
different OOP array libraries, neither of which is able to out-compete the
other. Meanwhile the rest of the world happily cooperates using pointers,
strides, CSR and CSC.
Now, there are limits to what you can do with strides and pointers. Noone's
denying the need for more. In my mind that's an API where you can do
fetch_block and put_block of cache-sized, N-dimensional blocks on an array;
but it might be something slightly different.
Here's what I'm asking: DO NOT simply keep extending ndarray and the NumPy C
API to deal with this issue.
What we need is duck-typing/polymorphism at the C level. If you keep
extending ndarray and the NumPy C API, what we'll have is a one-to-many
relationship: One provider of array technology, multiple consumers (with
hooks, I'm sure, but all implementations of the hook concept in the NumPy
world I've seen so far are a total disaster!).
What I think we need instead is something like PEP 3118 for the abstract
array that is only available block-wise with getters and setters. On the
Cython list we've decided that what we want for CEP 1000 (for boxing
callbacks etc.) is to extend PyTypeObject with our own fields; we could
create CEP 1001 to solve this issue and make any Python object an exporter
of block-getter/setter-arrays (better name needed).
What would be exported is (of course) a simple vtable:
typedef struct {
int (*get_block)(void *ctx, ssize_t *upper_left, ssize_t *lower_right,
...);
...
} block_getter_setter_array_vtable;
Let's please discuss the details *after* the fundamentals. But the reason I
put void* there instead of PyObject* is that I hope this could be used
beyond the Python world (say, Python-Julia); the void* would be handed to
you at the time you receive the vtable (however we handle that).
I suppose it would also be useful to have some way of predicting the
output format polymorphically for the caller. E.g. dense *
block_diagonal results in block diagonal, but dense + block_diagonal
results in dense, etc. It might be useful for the caller to know
whether it needs to allocate a sparse, dense or block-structured
array. Or maybe the polymorphic function could even do the allocation.
This needs to happen recursively of course, to avoid intermediate
temporaries. The compiler could easily handle that, and so could numpy
when it gets lazy evaluation.
Ah. But that depends too on the computation to be performed too; a)
elementwise, b) axis-wise reductions, c) linear algebra...
In my oomatrix code (please don't look at it, it's shameful) I do this
using multiple dispatch.
I'd rather ignore this for as long as we can, only implementing a[:] =
... -- I can't see how decisions here would trickle down to the API
that's used in the kernel, it's more like a pre-phase, and better
treated orthogonally.
I think if the heavy lifting of allocating output arrays and exporting
these arrays work in numpy, then support in Cython could use that (I
can already hear certain people object to more complicated array stuff
in Cython :). Even better here would be an external project that each
our projects could use (I still think the nditer sorting functionality
of arrays should be numpy-agnostic and externally available).
I agree with the separate project idea. It's trivial for NumPy to
incorporate that as one of its methods for exporting arrays, and I don't
think it makes sense to either build it into Cython, or outright depend
on NumPy.
Here's what I'd like (working title: NumBridge?).
- Mission: Be the double* + shape + strides in a world where that is
no longer enough, by providing tight, focused APIs/ABIs that are usable
across C/Fortran/Python.
I basically want something I can quickly acquire from a NumPy array,
then pass it into my C code without dragging along all the cruft that I
don't need.
- Written in pure C + specs, usable without Python
- PEP 3118 done right, basically semi-standardize the internal
Cython memoryview ABI and get something that's passable on stack
- Get block get/put API
- Iterator APIs
- Utility code for exporters and clients (iteration code, axis
reordering, etc.)
Is the scope of that insane,
Re: [Numpy-discussion] Fixing issue of future opaqueness of ndarray this summer
On Sat, May 12, 2012 at 3:55 PM, Dag Sverre Seljebotn
d.s.seljeb...@astro.uio.no wrote:
On 05/11/2012 03:37 PM, mark florisson wrote:
On 11 May 2012 12:13, Dag Sverre Seljebotnd.s.seljeb...@astro.uio.no
wrote:
(NumPy devs: I know, I get too many ideas. But this time I *really*
believe
in it, I think this is going to be *huge*. And if Mark F. likes it it's
not
going to be without manpower; and as his mentor I'd pitch in too here
and
there.)
(Mark F.: I believe this is *very* relevant to your GSoC. I certainly
don't
want to micro-manage your GSoC, just have your take.)
Travis, thank you very much for those good words in the NA-mask
interactions... thread. It put most of my concerns away. If anybody is
leaning towards for opaqueness because of its OOP purity, I want to
refer to
C++ and its walled-garden of ideological purity -- it has, what, 3-4
different OOP array libraries, neither of which is able to out-compete
the
other. Meanwhile the rest of the world happily cooperates using
pointers,
strides, CSR and CSC.
Now, there are limits to what you can do with strides and pointers.
Noone's
denying the need for more. In my mind that's an API where you can do
fetch_block and put_block of cache-sized, N-dimensional blocks on an
array;
but it might be something slightly different.
Here's what I'm asking: DO NOT simply keep extending ndarray and the
NumPy C
API to deal with this issue.
What we need is duck-typing/polymorphism at the C level. If you keep
extending ndarray and the NumPy C API, what we'll have is a one-to-many
relationship: One provider of array technology, multiple consumers (with
hooks, I'm sure, but all implementations of the hook concept in the
NumPy
world I've seen so far are a total disaster!).
What I think we need instead is something like PEP 3118 for the
abstract
array that is only available block-wise with getters and setters. On the
Cython list we've decided that what we want for CEP 1000 (for boxing
callbacks etc.) is to extend PyTypeObject with our own fields; we could
create CEP 1001 to solve this issue and make any Python object an
exporter
of block-getter/setter-arrays (better name needed).
What would be exported is (of course) a simple vtable:
typedef struct {
int (*get_block)(void *ctx, ssize_t *upper_left, ssize_t
*lower_right,
...);
...
} block_getter_setter_array_vtable;
Let's please discuss the details *after* the fundamentals. But the
reason I
put void* there instead of PyObject* is that I hope this could be used
beyond the Python world (say, Python-Julia); the void* would be
handed to
you at the time you receive the vtable (however we handle that).
I suppose it would also be useful to have some way of predicting the
output format polymorphically for the caller. E.g. dense *
block_diagonal results in block diagonal, but dense + block_diagonal
results in dense, etc. It might be useful for the caller to know
whether it needs to allocate a sparse, dense or block-structured
array. Or maybe the polymorphic function could even do the allocation.
This needs to happen recursively of course, to avoid intermediate
temporaries. The compiler could easily handle that, and so could numpy
when it gets lazy evaluation.
Ah. But that depends too on the computation to be performed too; a)
elementwise, b) axis-wise reductions, c) linear algebra...
In my oomatrix code (please don't look at it, it's shameful) I do this
using multiple dispatch.
I'd rather ignore this for as long as we can, only implementing a[:] =
... -- I can't see how decisions here would trickle down to the API
that's used in the kernel, it's more like a pre-phase, and better
treated orthogonally.
I think if the heavy lifting of allocating output arrays and exporting
these arrays work in numpy, then support in Cython could use that (I
can already hear certain people object to more complicated array stuff
in Cython :). Even better here would be an external project that each
our projects could use (I still think the nditer sorting functionality
of arrays should be numpy-agnostic and externally available).
I agree with the separate project idea. It's trivial for NumPy to
incorporate that as one of its methods for exporting arrays, and I don't
think it makes sense to either build it into Cython, or outright depend
on NumPy.
Here's what I'd like (working title: NumBridge?).
- Mission: Be the double* + shape + strides in a world where that is
no longer enough, by providing tight, focused APIs/ABIs that are usable
across C/Fortran/Python.
I basically want something I can quickly acquire from a NumPy array,
then pass it into my C code without dragging along all the cruft that I
don't need.
- Written in pure C + specs, usable without Python
- PEP 3118 done right, basically semi-standardize the internal
Cython memoryview ABI and get something that's
Re: [Numpy-discussion] Fixing issue of future opaqueness of ndarray this summer
Hi, On Sat, May 12, 2012 at 5:30 PM, Travis Oliphant tra...@continuum.io wrote: Your thoughts are definitely the future. We are currently building such a thing. We would like it to be open source. We are currently preparing a proposal to DARPA as part of their XDATA proposal in order to help fund this. Email me offlist if you would like to be a part of this proposal. You don't have to be a U.S. citizen to participate in this. What if this work that you are doing now is not open-source? What relationship will it have to numpy? It's difficult to have a productive discussion on the list if the main work is going on elsewhere, and we don't know what it is. See you, Matthew ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Fixing issue of future opaqueness of ndarray this summer
On May 12, 2012, at 9:01 PM, Matthew Brett wrote: Hi, On Sat, May 12, 2012 at 5:30 PM, Travis Oliphant tra...@continuum.io wrote: Your thoughts are definitely the future. We are currently building such a thing. We would like it to be open source.We are currently preparing a proposal to DARPA as part of their XDATA proposal in order to help fund this.Email me offlist if you would like to be a part of this proposal. You don't have to be a U.S. citizen to participate in this. What if this work that you are doing now is not open-source? What relationship will it have to numpy? Anything DARPA funds will be open source if we are lucky enough to get them to support it our vision. I'm not sure the exact relationship to current NumPy at this point.That's why I suggested the need to discuss this on a different venue than this list.But, this is just me personally interested in something at this point. It's difficult to have a productive discussion on the list if the main work is going on elsewhere, and we don't know what it is. I'm not sure how to make sense of this statement.The *main* work of *NumPy* is happening on this list. Nothing should be construed in what I have said to indicate otherwise. My statement that we are currently building such a thing obviously does not apply to NumPy. Of course work on other things might happen elsewhere and in other ways.At this point, all contributors to NumPy also work on other things as far as I know. Best, -Travis ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Fixing issue of future opaqueness of ndarray this summer
Hi, On Sat, May 12, 2012 at 7:24 PM, Travis Oliphant tra...@continuum.io wrote: On May 12, 2012, at 9:01 PM, Matthew Brett wrote: Hi, On Sat, May 12, 2012 at 5:30 PM, Travis Oliphant tra...@continuum.io wrote: Your thoughts are definitely the future. We are currently building such a thing. We would like it to be open source. We are currently preparing a proposal to DARPA as part of their XDATA proposal in order to help fund this. Email me offlist if you would like to be a part of this proposal. You don't have to be a U.S. citizen to participate in this. What if this work that you are doing now is not open-source? What relationship will it have to numpy? Anything DARPA funds will be open source if we are lucky enough to get them to support it our vision. I'm not sure the exact relationship to current NumPy at this point. That's why I suggested the need to discuss this on a different venue than this list. But, this is just me personally interested in something at this point. It's difficult to have a productive discussion on the list if the main work is going on elsewhere, and we don't know what it is. I'm not sure how to make sense of this statement. The *main* work of *NumPy* is happening on this list. Nothing should be construed in what I have said to indicate otherwise. My statement that we are currently building such a thing obviously does not apply to NumPy. Of course work on other things might happen elsewhere and in other ways. At this point, all contributors to NumPy also work on other things as far as I know. In your email above you said Your thoughts are definitely the future. We are currently building such a thing. We would like it to be open source. I assume you meant, the future of numpy. Did you mean something else? Will there be a 'numpy-pro'? If so, how would that relate to the future of numpy? See you, Matthew ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Fixing issue of future opaqueness of ndarray this summer
On May 12, 2012, at 9:39 PM, Matthew Brett wrote: Hi, On Sat, May 12, 2012 at 7:24 PM, Travis Oliphant tra...@continuum.io wrote: On May 12, 2012, at 9:01 PM, Matthew Brett wrote: Hi, On Sat, May 12, 2012 at 5:30 PM, Travis Oliphant tra...@continuum.io wrote: Your thoughts are definitely the future. We are currently building such a thing. We would like it to be open source.We are currently preparing a proposal to DARPA as part of their XDATA proposal in order to help fund this.Email me offlist if you would like to be a part of this proposal.You don't have to be a U.S. citizen to participate in this. What if this work that you are doing now is not open-source? What relationship will it have to numpy? Anything DARPA funds will be open source if we are lucky enough to get them to support it our vision. I'm not sure the exact relationship to current NumPy at this point.That's why I suggested the need to discuss this on a different venue than this list.But, this is just me personally interested in something at this point. It's difficult to have a productive discussion on the list if the main work is going on elsewhere, and we don't know what it is. I'm not sure how to make sense of this statement.The *main* work of *NumPy* is happening on this list. Nothing should be construed in what I have said to indicate otherwise. My statement that we are currently building such a thing obviously does not apply to NumPy. Of course work on other things might happen elsewhere and in other ways. At this point, all contributors to NumPy also work on other things as far as I know. In your email above you said Your thoughts are definitely the future. We are currently building such a thing. We would like it to be open source. I assume you meant, the future of numpy. Did you mean something else? Will there be a 'numpy-pro'? If so, how would that relate to the future of numpy? I think I've clarified that I meant something else in that context. Any statement about any rumored NumPy-Pro is unrelated and completely off-topic (and inappropriate to discuss on this list). off-topic Of course there could be such a thing. Several people have produced things like that before including Enthought (with it's MKL-linked NumPy) and Interactive SuperComputing (starp). I think it would be great for NumPy if there were a lot of such offerings. My preference for such proprietary products is that they eventually become open source which should be pretty clear to anyone paying attention to the way I sold my book Guide to NumPy --- which I might add is currently freely available and became a large primary source for the NumPy Documentation project that Joe Harrington spearheaded. /off topic But such discussions are really not appropriate for this list. I would really hope that we can keep the NumPy discussion list on topic to NumPy. Obviously there will be the occasional announcement that describe related but extra-list topics (like Dag posts about Cython and others post about SymPy or SciPy or NumFOCUS, or, conferences, or whatever). But they should not really create threads of discussion. A great place for such general discussions (and perhaps discussions about governance or process) is probably a NumFOCUS mailing list. NumFOCUS is trying to promote the Open Source tools generally and is in very early stages of organization with the by-laws not even written yet.I suspect that the board of NumFOCUS would be excited for people to get involved and help organize and participate in forums and mailings lists that discuss how to help the open source scientific stack for Python progress. The best way to get involved there is to go to www.numfocus.org and email i...@numfocus.org Best, -Travis ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] Fixing issue of future opaqueness of ndarray this summer
(NumPy devs: I know, I get too many ideas. But this time I *really*
believe in it, I think this is going to be *huge*. And if Mark F. likes
it it's not going to be without manpower; and as his mentor I'd pitch in
too here and there.)
(Mark F.: I believe this is *very* relevant to your GSoC. I certainly
don't want to micro-manage your GSoC, just have your take.)
Travis, thank you very much for those good words in the NA-mask
interactions... thread. It put most of my concerns away. If anybody is
leaning towards for opaqueness because of its OOP purity, I want to
refer to C++ and its walled-garden of ideological purity -- it has,
what, 3-4 different OOP array libraries, neither of which is able to
out-compete the other. Meanwhile the rest of the world happily
cooperates using pointers, strides, CSR and CSC.
Now, there are limits to what you can do with strides and pointers.
Noone's denying the need for more. In my mind that's an API where you
can do fetch_block and put_block of cache-sized, N-dimensional blocks on
an array; but it might be something slightly different.
Here's what I'm asking: DO NOT simply keep extending ndarray and the
NumPy C API to deal with this issue.
What we need is duck-typing/polymorphism at the C level. If you keep
extending ndarray and the NumPy C API, what we'll have is a one-to-many
relationship: One provider of array technology, multiple consumers (with
hooks, I'm sure, but all implementations of the hook concept in the
NumPy world I've seen so far are a total disaster!).
What I think we need instead is something like PEP 3118 for the
abstract array that is only available block-wise with getters and
setters. On the Cython list we've decided that what we want for CEP 1000
(for boxing callbacks etc.) is to extend PyTypeObject with our own
fields; we could create CEP 1001 to solve this issue and make any Python
object an exporter of block-getter/setter-arrays (better name needed).
What would be exported is (of course) a simple vtable:
typedef struct {
int (*get_block)(void *ctx, ssize_t *upper_left, ssize_t
*lower_right, ...);
...
} block_getter_setter_array_vtable;
Let's please discuss the details *after* the fundamentals. But the
reason I put void* there instead of PyObject* is that I hope this could
be used beyond the Python world (say, Python-Julia); the void* would
be handed to you at the time you receive the vtable (however we handle
that).
I think this would fit neatly in Mark F.'s GSoC (Mark F.?), because you
could embed the block-transposition that's needed for efficient arr +
arr.T at this level.
Imagine being able to do this in Cython:
a[...] = b + c * d
and have that essentially compile to the numexpr blocked approach, *but*
where b, c, and d can have whatever type that exports CEP 1001? So c
could be a diagonal array which uses O(n) storage to export O(n^2)
elements, for instance, and the unrolled Cython code never needs to know.
As far as NumPy goes, something along these lines should hopefully mean
that new C code being written doesn't rely so much on what exactly goes
into ndarray and what goes into other classes; so that we don't get
the same problem again that we do now with code that doesn't use PEP 3118.
Dag
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Fixing issue of future opaqueness of ndarray this summer
On 05/11/2012 01:13 PM, Dag Sverre Seljebotn wrote:
(NumPy devs: I know, I get too many ideas. But this time I *really*
believe in it, I think this is going to be *huge*. And if Mark F. likes
it it's not going to be without manpower; and as his mentor I'd pitch in
too here and there.)
(Mark F.: I believe this is *very* relevant to your GSoC. I certainly
don't want to micro-manage your GSoC, just have your take.)
For the information of the rest of you:
http://www.google-melange.com/gsoc/project/google/gsoc2012/markflorisson88/30002
Dag
Travis, thank you very much for those good words in the NA-mask
interactions... thread. It put most of my concerns away. If anybody is
leaning towards for opaqueness because of its OOP purity, I want to
refer to C++ and its walled-garden of ideological purity -- it has,
what, 3-4 different OOP array libraries, neither of which is able to
out-compete the other. Meanwhile the rest of the world happily
cooperates using pointers, strides, CSR and CSC.
Now, there are limits to what you can do with strides and pointers.
Noone's denying the need for more. In my mind that's an API where you
can do fetch_block and put_block of cache-sized, N-dimensional blocks on
an array; but it might be something slightly different.
Here's what I'm asking: DO NOT simply keep extending ndarray and the
NumPy C API to deal with this issue.
What we need is duck-typing/polymorphism at the C level. If you keep
extending ndarray and the NumPy C API, what we'll have is a one-to-many
relationship: One provider of array technology, multiple consumers (with
hooks, I'm sure, but all implementations of the hook concept in the
NumPy world I've seen so far are a total disaster!).
What I think we need instead is something like PEP 3118 for the
abstract array that is only available block-wise with getters and
setters. On the Cython list we've decided that what we want for CEP 1000
(for boxing callbacks etc.) is to extend PyTypeObject with our own
fields; we could create CEP 1001 to solve this issue and make any Python
object an exporter of block-getter/setter-arrays (better name needed).
What would be exported is (of course) a simple vtable:
typedef struct {
int (*get_block)(void *ctx, ssize_t *upper_left, ssize_t
*lower_right, ...);
...
} block_getter_setter_array_vtable;
Let's please discuss the details *after* the fundamentals. But the
reason I put void* there instead of PyObject* is that I hope this could
be used beyond the Python world (say, Python-Julia); the void* would
be handed to you at the time you receive the vtable (however we handle
that).
I think this would fit neatly in Mark F.'s GSoC (Mark F.?), because you
could embed the block-transposition that's needed for efficient arr +
arr.T at this level.
Imagine being able to do this in Cython:
a[...] = b + c * d
and have that essentially compile to the numexpr blocked approach, *but*
where b, c, and d can have whatever type that exports CEP 1001? So c
could be a diagonal array which uses O(n) storage to export O(n^2)
elements, for instance, and the unrolled Cython code never needs to know.
As far as NumPy goes, something along these lines should hopefully mean
that new C code being written doesn't rely so much on what exactly goes
into ndarray and what goes into other classes; so that we don't get
the same problem again that we do now with code that doesn't use PEP 3118.
Dag
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Fixing issue of future opaqueness of ndarray this summer
On 11 May 2012 12:13, Dag Sverre Seljebotn d.s.seljeb...@astro.uio.no wrote:
(NumPy devs: I know, I get too many ideas. But this time I *really* believe
in it, I think this is going to be *huge*. And if Mark F. likes it it's not
going to be without manpower; and as his mentor I'd pitch in too here and
there.)
(Mark F.: I believe this is *very* relevant to your GSoC. I certainly don't
want to micro-manage your GSoC, just have your take.)
Travis, thank you very much for those good words in the NA-mask
interactions... thread. It put most of my concerns away. If anybody is
leaning towards for opaqueness because of its OOP purity, I want to refer to
C++ and its walled-garden of ideological purity -- it has, what, 3-4
different OOP array libraries, neither of which is able to out-compete the
other. Meanwhile the rest of the world happily cooperates using pointers,
strides, CSR and CSC.
Now, there are limits to what you can do with strides and pointers. Noone's
denying the need for more. In my mind that's an API where you can do
fetch_block and put_block of cache-sized, N-dimensional blocks on an array;
but it might be something slightly different.
Here's what I'm asking: DO NOT simply keep extending ndarray and the NumPy C
API to deal with this issue.
What we need is duck-typing/polymorphism at the C level. If you keep
extending ndarray and the NumPy C API, what we'll have is a one-to-many
relationship: One provider of array technology, multiple consumers (with
hooks, I'm sure, but all implementations of the hook concept in the NumPy
world I've seen so far are a total disaster!).
What I think we need instead is something like PEP 3118 for the abstract
array that is only available block-wise with getters and setters. On the
Cython list we've decided that what we want for CEP 1000 (for boxing
callbacks etc.) is to extend PyTypeObject with our own fields; we could
create CEP 1001 to solve this issue and make any Python object an exporter
of block-getter/setter-arrays (better name needed).
What would be exported is (of course) a simple vtable:
typedef struct {
int (*get_block)(void *ctx, ssize_t *upper_left, ssize_t *lower_right,
...);
...
} block_getter_setter_array_vtable;
Interesting idea, I think I like it. So I suppose these blocks could
even be used in a sparse context, where it returns one-sized blocks
for un-surrounded elements. This means returned blocks can always be
variably sized, correct? So will each block be associated with an
index space? Otherwise, how do we know the element-wise
correspondence? Will there be a default element for missing
(unreturned) values?
Am I correct in assuming elements in returned blocks always return
elements in element-wise order? In case you manually have to block
other parts of an array (lets say it's stored in Fortran order), then
the variable sized block nature may be complicating the tiling
pattern.
Let's please discuss the details *after* the fundamentals. But the reason I
put void* there instead of PyObject* is that I hope this could be used
beyond the Python world (say, Python-Julia); the void* would be handed to
you at the time you receive the vtable (however we handle that).
Yes, we should definitely not stick to objects here.
I think this would fit neatly in Mark F.'s GSoC (Mark F.?), because you
could embed the block-transposition that's needed for efficient arr +
arr.T at this level.
Imagine being able to do this in Cython:
a[...] = b + c * d
and have that essentially compile to the numexpr blocked approach, *but*
where b, c, and d can have whatever type that exports CEP 1001? So c could
be a diagonal array which uses O(n) storage to export O(n^2) elements, for
instance, and the unrolled Cython code never needs to know.
I assume random accesses will happen through some kind of
B-tree/sparse index like mechanism?
As far as NumPy goes, something along these lines should hopefully mean that
new C code being written doesn't rely so much on what exactly goes into
ndarray and what goes into other classes; so that we don't get the same
problem again that we do now with code that doesn't use PEP 3118.
Dag
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Fixing issue of future opaqueness of ndarray this summer
On 11 May 2012 12:13, Dag Sverre Seljebotn d.s.seljeb...@astro.uio.no wrote:
(NumPy devs: I know, I get too many ideas. But this time I *really* believe
in it, I think this is going to be *huge*. And if Mark F. likes it it's not
going to be without manpower; and as his mentor I'd pitch in too here and
there.)
(Mark F.: I believe this is *very* relevant to your GSoC. I certainly don't
want to micro-manage your GSoC, just have your take.)
Travis, thank you very much for those good words in the NA-mask
interactions... thread. It put most of my concerns away. If anybody is
leaning towards for opaqueness because of its OOP purity, I want to refer to
C++ and its walled-garden of ideological purity -- it has, what, 3-4
different OOP array libraries, neither of which is able to out-compete the
other. Meanwhile the rest of the world happily cooperates using pointers,
strides, CSR and CSC.
Now, there are limits to what you can do with strides and pointers. Noone's
denying the need for more. In my mind that's an API where you can do
fetch_block and put_block of cache-sized, N-dimensional blocks on an array;
but it might be something slightly different.
Here's what I'm asking: DO NOT simply keep extending ndarray and the NumPy C
API to deal with this issue.
What we need is duck-typing/polymorphism at the C level. If you keep
extending ndarray and the NumPy C API, what we'll have is a one-to-many
relationship: One provider of array technology, multiple consumers (with
hooks, I'm sure, but all implementations of the hook concept in the NumPy
world I've seen so far are a total disaster!).
What I think we need instead is something like PEP 3118 for the abstract
array that is only available block-wise with getters and setters. On the
Cython list we've decided that what we want for CEP 1000 (for boxing
callbacks etc.) is to extend PyTypeObject with our own fields; we could
create CEP 1001 to solve this issue and make any Python object an exporter
of block-getter/setter-arrays (better name needed).
What would be exported is (of course) a simple vtable:
typedef struct {
int (*get_block)(void *ctx, ssize_t *upper_left, ssize_t *lower_right,
...);
...
} block_getter_setter_array_vtable;
Let's please discuss the details *after* the fundamentals. But the reason I
put void* there instead of PyObject* is that I hope this could be used
beyond the Python world (say, Python-Julia); the void* would be handed to
you at the time you receive the vtable (however we handle that).
I suppose it would also be useful to have some way of predicting the
output format polymorphically for the caller. E.g. dense *
block_diagonal results in block diagonal, but dense + block_diagonal
results in dense, etc. It might be useful for the caller to know
whether it needs to allocate a sparse, dense or block-structured
array. Or maybe the polymorphic function could even do the allocation.
This needs to happen recursively of course, to avoid intermediate
temporaries. The compiler could easily handle that, and so could numpy
when it gets lazy evaluation.
I think if the heavy lifting of allocating output arrays and exporting
these arrays work in numpy, then support in Cython could use that (I
can already hear certain people object to more complicated array stuff
in Cython :). Even better here would be an external project that each
our projects could use (I still think the nditer sorting functionality
of arrays should be numpy-agnostic and externally available).
I think this would fit neatly in Mark F.'s GSoC (Mark F.?), because you
could embed the block-transposition that's needed for efficient arr +
arr.T at this level.
Imagine being able to do this in Cython:
a[...] = b + c * d
and have that essentially compile to the numexpr blocked approach, *but*
where b, c, and d can have whatever type that exports CEP 1001? So c could
be a diagonal array which uses O(n) storage to export O(n^2) elements, for
instance, and the unrolled Cython code never needs to know.
As far as NumPy goes, something along these lines should hopefully mean that
new C code being written doesn't rely so much on what exactly goes into
ndarray and what goes into other classes; so that we don't get the same
problem again that we do now with code that doesn't use PEP 3118.
Dag
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Fixing issue of future opaqueness of ndarray this summer
On Fri, May 11, 2012 at 6:13 AM, Dag Sverre Seljebotn
d.s.seljeb...@astro.uio.no wrote:
(NumPy devs: I know, I get too many ideas. But this time I *really*
believe in it, I think this is going to be *huge*. And if Mark F. likes
it it's not going to be without manpower; and as his mentor I'd pitch in
too here and there.)
(Mark F.: I believe this is *very* relevant to your GSoC. I certainly
don't want to micro-manage your GSoC, just have your take.)
Travis, thank you very much for those good words in the NA-mask
interactions... thread. It put most of my concerns away. If anybody is
leaning towards for opaqueness because of its OOP purity, I want to
refer to C++ and its walled-garden of ideological purity -- it has,
what, 3-4 different OOP array libraries, neither of which is able to
out-compete the other. Meanwhile the rest of the world happily
cooperates using pointers, strides, CSR and CSC.
Now, there are limits to what you can do with strides and pointers.
Noone's denying the need for more. In my mind that's an API where you
can do fetch_block and put_block of cache-sized, N-dimensional blocks on
an array; but it might be something slightly different.
Here's what I'm asking: DO NOT simply keep extending ndarray and the
NumPy C API to deal with this issue.
What we need is duck-typing/polymorphism at the C level. If you keep
extending ndarray and the NumPy C API, what we'll have is a one-to-many
relationship: One provider of array technology, multiple consumers (with
hooks, I'm sure, but all implementations of the hook concept in the
NumPy world I've seen so far are a total disaster!).
There is similar intent behind an idea I raised last summer here:
http://mail.scipy.org/pipermail/numpy-discussion/2011-June/056945.html
I stopped doing anything on it when considering the scope of it with the
linear algebra functions included like Pauli suggested. Nathaniel did an
experimental implementation of some parts of the idea in Python here:
https://github.com/njsmith/numpyNEP/blob/master/numpyNEP.py#L107
What I think we need instead is something like PEP 3118 for the
abstract array that is only available block-wise with getters and
setters. On the Cython list we've decided that what we want for CEP 1000
(for boxing callbacks etc.) is to extend PyTypeObject with our own
fields; we could create CEP 1001 to solve this issue and make any Python
object an exporter of block-getter/setter-arrays (better name needed).
What would be exported is (of course) a simple vtable:
typedef struct {
int (*get_block)(void *ctx, ssize_t *upper_left, ssize_t
*lower_right, ...);
...
} block_getter_setter_array_vtable;
Let's please discuss the details *after* the fundamentals. But the
reason I put void* there instead of PyObject* is that I hope this could
be used beyond the Python world (say, Python-Julia); the void* would
be handed to you at the time you receive the vtable (however we handle
that).
I think this would fit neatly in Mark F.'s GSoC (Mark F.?), because you
could embed the block-transposition that's needed for efficient arr +
arr.T at this level.
Imagine being able to do this in Cython:
a[...] = b + c * d
and have that essentially compile to the numexpr blocked approach, *but*
where b, c, and d can have whatever type that exports CEP 1001? So c
could be a diagonal array which uses O(n) storage to export O(n^2)
elements, for instance, and the unrolled Cython code never needs to know.
As far as NumPy goes, something along these lines should hopefully mean
that new C code being written doesn't rely so much on what exactly goes
into ndarray and what goes into other classes; so that we don't get
the same problem again that we do now with code that doesn't use PEP 3118.
This general idea is very good. I think PEP 3118 captures a lot of the
essence of the ndarray, but there's a lot of potential generality that it
doesn't handle, such as the diagonal array or pluggable dtypes.
-Mark
Dag
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Fixing issue of future opaqueness of ndarray this summer
On Fri, May 11, 2012 at 8:37 AM, mark florisson
markflorisso...@gmail.comwrote:
On 11 May 2012 12:13, Dag Sverre Seljebotn d.s.seljeb...@astro.uio.no
wrote:
(NumPy devs: I know, I get too many ideas. But this time I *really*
believe
in it, I think this is going to be *huge*. And if Mark F. likes it it's
not
going to be without manpower; and as his mentor I'd pitch in too here and
there.)
(Mark F.: I believe this is *very* relevant to your GSoC. I certainly
don't
want to micro-manage your GSoC, just have your take.)
Travis, thank you very much for those good words in the NA-mask
interactions... thread. It put most of my concerns away. If anybody is
leaning towards for opaqueness because of its OOP purity, I want to
refer to
C++ and its walled-garden of ideological purity -- it has, what, 3-4
different OOP array libraries, neither of which is able to out-compete
the
other. Meanwhile the rest of the world happily cooperates using pointers,
strides, CSR and CSC.
Now, there are limits to what you can do with strides and pointers.
Noone's
denying the need for more. In my mind that's an API where you can do
fetch_block and put_block of cache-sized, N-dimensional blocks on an
array;
but it might be something slightly different.
Here's what I'm asking: DO NOT simply keep extending ndarray and the
NumPy C
API to deal with this issue.
What we need is duck-typing/polymorphism at the C level. If you keep
extending ndarray and the NumPy C API, what we'll have is a one-to-many
relationship: One provider of array technology, multiple consumers (with
hooks, I'm sure, but all implementations of the hook concept in the NumPy
world I've seen so far are a total disaster!).
What I think we need instead is something like PEP 3118 for the
abstract
array that is only available block-wise with getters and setters. On the
Cython list we've decided that what we want for CEP 1000 (for boxing
callbacks etc.) is to extend PyTypeObject with our own fields; we could
create CEP 1001 to solve this issue and make any Python object an
exporter
of block-getter/setter-arrays (better name needed).
What would be exported is (of course) a simple vtable:
typedef struct {
int (*get_block)(void *ctx, ssize_t *upper_left, ssize_t *lower_right,
...);
...
} block_getter_setter_array_vtable;
Let's please discuss the details *after* the fundamentals. But the
reason I
put void* there instead of PyObject* is that I hope this could be used
beyond the Python world (say, Python-Julia); the void* would be handed
to
you at the time you receive the vtable (however we handle that).
I suppose it would also be useful to have some way of predicting the
output format polymorphically for the caller. E.g. dense *
block_diagonal results in block diagonal, but dense + block_diagonal
results in dense, etc. It might be useful for the caller to know
whether it needs to allocate a sparse, dense or block-structured
array. Or maybe the polymorphic function could even do the allocation.
This needs to happen recursively of course, to avoid intermediate
temporaries. The compiler could easily handle that, and so could numpy
when it gets lazy evaluation.
I think if the heavy lifting of allocating output arrays and exporting
these arrays work in numpy, then support in Cython could use that (I
can already hear certain people object to more complicated array stuff
in Cython :). Even better here would be an external project that each
our projects could use (I still think the nditer sorting functionality
of arrays should be numpy-agnostic and externally available).
It might be nice to expose something which gives an nditer-style looping
primitive through the CEP 1001 mechanism. I could imagine a pure C version
of this and an LLVM bitcode version which could inline into numba or other
LLVM producing systems.
-Mark
I think this would fit neatly in Mark F.'s GSoC (Mark F.?), because you
could embed the block-transposition that's needed for efficient arr +
arr.T at this level.
Imagine being able to do this in Cython:
a[...] = b + c * d
and have that essentially compile to the numexpr blocked approach, *but*
where b, c, and d can have whatever type that exports CEP 1001? So c
could
be a diagonal array which uses O(n) storage to export O(n^2) elements,
for
instance, and the unrolled Cython code never needs to know.
As far as NumPy goes, something along these lines should hopefully mean
that
new C code being written doesn't rely so much on what exactly goes into
ndarray and what goes into other classes; so that we don't get the same
problem again that we do now with code that doesn't use PEP 3118.
Dag
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
___
