sjperkins commented on PR #34483:
URL: https://github.com/apache/arrow/pull/34483#issuecomment-1491797111
> I am not a huge fan of the idea of creating classes on the fly .. Also,
does this give something more useful than wrapping it just in a base class as
is done now? (because right now this generated class also doesn't have any
extension-type specific logic?)
Yes, I understand the suggestion is a bit exotic. I think what it gives is
the ability to associate specific Python classes with C++ Extension
Types/Arrays and attach (mainly) `{from,to}_numpy` functionality to the Python
class. There probably are nicer ways to do it, but nothing comes immediately to
mind. Perhaps there's a way to do this with the existing C++ PyExtensionType
machinery.
> Yeah, so if we have a way to register a python class to use as the type
class for an extension type implemented in C++ (#33997), you can override the
`to_numpy()` method. However, than only works if you call this method directly
on the array object. But if you convert a table, or a chunked array, it will
still go through the C++ layer which currently falls back on converting the
storage array. So we might need to think about a more general mechanism here to
tap into this conversion logic.
Thanks for pointing this out, this gives me a broader perspective on what
would be useful.
>
> > This may also be relevant to #8510 which, from imperfect memory, uses
FixedSizeListArray's to represent Tensors.
>
> Yes, that one is close to being merged, and then we can expose this in
python, which might be a useful exercise to see how this goes / what we can
learn from this (but of course it's an internal one, so we can hard code
support for it)
I'll think about this a bit more. let me know if there're approaches you
think would be useful to experiment with.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
