On Sep 30, 2009, at 10:34 AM, Christopher Barker wrote: > Magnus Lie Hetland wrote: >> Yeah, one-dimensional data -- simply a list of ints. >> >> I'll just go for a Python list for now, then, and look into it if >> performance becomes a problem. (It's not going to be a long list, but >> it may be traversed hundreds of thousands of times, so who >> knows... :->) > > Funny you should write this. I was just composing a note in my head > about a proposal for a numpy "accumulating array". This idea was > inspired by a discussion at SciPY, in which we spent a LOT of time > during the numpy tutorial talking about how to accumulate values in an > array when you don't know how big the array needs to be when you > start. > > The "standard practice" is to accumulate in a python list, then > convert > the final result into an array. This is a good idea because Python > lists > are standard, well tested, efficient, etc. > > However, as was pointed out in that lengthy discussion, if what you > are > doing is accumulating is a whole bunch of numbers (ints, floats, > whatever), or particularly if you need to accumulate a data type that > plain python doesn't support, there is a lot of overhead involved: a > python float type is pretty heavyweight. If performance or memory > use is > important, it might create issues. > > What I have in mind is very simple. It would be: > - Only 1-d > - Support append() and extend() methods > - Support any valid numpy dtype > - which could even get you pseudo n-d arrays... > - maybe it would act like an array in other ways, I'm not so sure.. > > > It could be written in pure python, using np.resize()(or concatenate, > or,...), and certainly written efficiently in Cython or even (no!) C.
Sounds useful. One thing that would be missing from a pure Python implementation is the ability to append and get raw c types, rather than just their wrapped counterparts, which would probably be the bulk of the overhead. (In Cython one would have to write a separate class for each type, oh if only we have templates already...) - Robert _______________________________________________ Cython-dev mailing list [email protected] http://codespeak.net/mailman/listinfo/cython-dev
