Does the new DType system in NumPy 2 make something like this more possible? I would suspect that the user would have to write a lot of code to have reasonable performance if it was.
Kevin On Wed, Mar 13, 2024 at 3:55 PM Nathan <nathan.goldb...@gmail.com> wrote: > Yes, an array of references still has a fixed size width in the array > buffer. You can think of each entry in the array as a pointer to some other > memory on the heap, which can be a dynamic memory allocation. > > There's no way in NumPy to support variable-sized array elements in the > array buffer, since that assumption is key to how numpy implements strided > ufuncs and broadcasting., > > On Wed, Mar 13, 2024 at 9:34 AM Dom Grigonis <dom.grigo...@gmail.com> > wrote: > >> Thank you for this. >> >> I am just starting to think about these things, so I appreciate your >> patience. >> >> But isn’t it still true that all elements of an array are still of the >> same size in memory? >> >> I am thinking along the lines of per-element dynamic memory management. >> Such that if I had array [0, 1e10000], the first element would default to >> reasonably small size in memory. >> >> On 13 Mar 2024, at 16:29, Nathan <nathan.goldb...@gmail.com> wrote: >> >> It is possible to do this using the new DType system. >> >> Sebastian wrote a sketch for a DType backed by the GNU multiprecision >> float library: >> https://github.com/numpy/numpy-user-dtypes/tree/main/mpfdtype >> >> It adds a significant amount of complexity to store data outside the >> array buffer and introduces the possibility of use-after-free and dangling >> reference errors that are impossible if the array does not use embedded >> references, so that’s the main reason it hasn’t been done much. >> >> On Wed, Mar 13, 2024 at 8:17 AM Dom Grigonis <dom.grigo...@gmail.com> >> wrote: >> >>> Hi all, >>> >>> Say python’s builtin `int` type. It can be as large as memory allows. >>> >>> np.ndarray on the other hand is optimized for vectorization via strides, >>> memory structure and many things that I probably don’t know. Well the point >>> is that it is convenient and efficient to use for many things in comparison >>> to python’s built-in list of integers. >>> >>> So, I am thinking whether something in between exists? (And obviously >>> something more clever than np.array(dtype=object)) >>> >>> Probably something similar to `StringDType`, but for integers and >>> floats. (It’s just my guess. I don’t know anything about `StringDType`, >>> but just guessing it must be better than np.array(dtype=object) in >>> combination with np.vectorize) >>> >>> Regards, >>> dgpb >>> >>> _______________________________________________ >>> NumPy-Discussion mailing list -- numpy-discussion@python.org >>> To unsubscribe send an email to numpy-discussion-le...@python.org >>> https://mail.python.org/mailman3/lists/numpy-discussion.python.org/ >>> Member address: nathan12...@gmail.com >>> >> _______________________________________________ >> NumPy-Discussion mailing list -- numpy-discussion@python.org >> To unsubscribe send an email to numpy-discussion-le...@python.org >> https://mail.python.org/mailman3/lists/numpy-discussion.python.org/ >> Member address: dom.grigo...@gmail.com >> >> >> _______________________________________________ >> NumPy-Discussion mailing list -- numpy-discussion@python.org >> To unsubscribe send an email to numpy-discussion-le...@python.org >> https://mail.python.org/mailman3/lists/numpy-discussion.python.org/ >> Member address: nathan12...@gmail.com >> > _______________________________________________ > NumPy-Discussion mailing list -- numpy-discussion@python.org > To unsubscribe send an email to numpy-discussion-le...@python.org > https://mail.python.org/mailman3/lists/numpy-discussion.python.org/ > Member address: kevin.k.shepp...@gmail.com >
_______________________________________________ NumPy-Discussion mailing list -- numpy-discussion@python.org To unsubscribe send an email to numpy-discussion-le...@python.org https://mail.python.org/mailman3/lists/numpy-discussion.python.org/ Member address: arch...@mail-archive.com
