paleolimbot commented on issue #38717: URL: https://github.com/apache/arrow/issues/38717#issuecomment-1811704310
Yes, I think this needs C++-level support (but I think it's worth it!). I will try to take a stab at an implementation before 15.0.0...I would like to use it in the R bindings as well. To summarise where this came up in GeoArrow-land...we'd like to base our Python ecosystem on the C data/stream interfaces (e.g., functions use the dunder methods when consuming input, and return an object that implements a dunder); however, right now we can't consume or return ChunkedArray. We could force a concatenation (maybe expensive and maybe resulting in overflow since one common extension type is based on binary), or we could constantly special-case the ChunkedArray and loop over chunks at the Python level (verbose, and performance degrades when chunks get small/numerous). The ChunkedArray is the most common Array representation in pyarrow (e.g., column in a Table!), so this comes up quite a bit. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
