Just pinging on this thread to hopefully encourage more comments and engagement on the document. I still have to respond to a few of Antoine's open comments, but so far there's only be the one individual who has given feedback.
I've added a large "background context" section at the top of the document to hopefully make it easier for individuals to understand the motivation and comment on the document. If there's no objections by the end of the week I'd like to propose a vote to officially adopt this and start putting a PR together for updating the docs. But personally I'd rather see more comments than it being adopted by default of no one objecting. --Matt On Wed, Feb 7, 2024, 11:50 AM Jean-Baptiste Onofré <j...@nanthrax.net> wrote: > Hi Matt > > Thanks for sharing. It looks interesting and I like the idea. > > Let me review the document and eventually add comments. > > Thanks ! > Regards > JB > > On Sat, Feb 3, 2024 at 12:22 AM Matt Topol <zotthewiz...@gmail.com> wrote: > > > > Hey all, > > > > In my current work I've been experimenting and playing around with > > utilizing Arrow and non-cpu memory data. While the creation of the > > ArrowDeviceArray struct and the enhancements to the Arrow library Device > > abstractions were necessary, there is also a need to extend the > > communications specs we utilize, i.e. Flight. > > > > Currently there is no real way to utilize Arrow Flight with shared memory > > or with non-CPU memory (without an expensive Device -> Host copy first). > To > > this end I've done a bunch of research and toying around and came up > with a > > protocol to propose and a reference implementation using UCX[1]. Attached > > to the proposal is also a couple extensions for Flight itself to make it > > easier for users to still use Flight for metadata / dataset information > and > > then point consumers elsewhere to actually retrieve the data. The idea > here > > is that this would be a new specification for how to transport Arrow data > > across these high-performance transports such as UCX / libfabric / shared > > memory / etc. We wouldn't necessarily expose / directly add > implementations > > of the spec to the Arrow libraries, just provide reference/example > > implementations. > > > > I've written the proposal up on a google doc[2] that everyone should be > > able to comment on. Once we get some community discussion on there, if > > everyone is okay with it I'd like eventually do a vote on adopting this > > spec and if we do, I'll then make a PR to start adding it to the Arrow > > documentation, etc. > > > > Anyways, thank you everyone in advance for your feedback and comments! > > > > --Matt > > > > [1]: https://github.com/openucx/ucx/ > > [2]: > > > https://docs.google.com/document/d/1zHbnyK1r6KHpMOtEdIg1EZKNzHx-MVgUMOzB87GuXyk/edit?usp=sharing >
