Matt, (See my reply to Antoine for some clarifications about famfs that may or may not have been obvious).
In current-day workloads sharding and reshuffling are unavoidable, and will probably never become fully avoidable. Your Arrow Communication Extensions look to me like sensible extensions to address some of the pain associated with shuffling data around. Do I that about right, or am I missing things? The big idea with famfs is that any data placed in famfs-shared- memory need not be reshuffled because everybody can see the same namespace to the same data (hey, a file system is a name space). In situations where shared memory is available, this can be a big win. But even assuming shared memory becomes common, not all workloads will have access to it. I would love to meet and brainstorm with interested parties about this problem space - perhaps it could lead to a working group - and I very much hope it would lead to compelling POC ideas. We are currently reviving some POC work that was done a few months ago; I should be able to share some of that in the next few weeks. Is anybody interested in meeting to explore this problem space? Best regards, John Groves Micron On 24/04/10 11:32AM, Matt Topol wrote: > Hi John, > > I recently proposed on the mailing list an experimental extension of the > Arrow IPC protocol that would make it easier to leverage disaggregated > shared memory along with non-cpu memory via utilities such as UCX and > libfabric [1]. I'll be putting together a more formal description of it > that will be being added to the Arrow website, but would still love any > feedback you might have for it and how it could work with Famfs. It would > be interesting if we could put together a POC leveraging CXL, Famfs, and > the proposed protocol. There were a few suggestions and responses to the > document that included use-cases from interested individuals such as being > able to stick an Arrow record batch into shared memory and update it > in-place. > > I think the best way to get the word out and bring interest to this would > be to create a really great POC showing a particular use-case being > significantly improved by utilizing all of these pieces. I'd love to see if > we could work together and construct such a POC and get some benchmarking > numbers for comparisons. A great place to start would be comparing the > disaggregated shared memory performance against various network transfer / > RPC protocols for some use cases processing large amounts of data in a > distributed system. > > What do you think? > > --Matt > > [1]: > https://docs.google.com/document/d/1zHbnyK1r6KHpMOtEdIg1EZKNzHx-MVgUMOzB87GuXyk/edit#heading=h.38515dnp2bdb > > On Tue, Apr 9, 2024 at 10:14 PM John Groves <j...@groves.net> wrote: > > > This is a request for comments from the Arrow developer community. > > > > I’m reaching out to start making the Arrow community aware of work that my > > team at Micron has recently open-sourced. Because of the Compute Express > > Link (CXL) standard, sharable disaggregated memory is coming – this is > > memory shared by multiple nodes in a cluster. Arrow and the other > > zero-copy formats are a great fit for shared memory if a natural enough > > access method emerges. > > > > That’s where famfs comes in. (Famfs stands for Fabric-Attached Memory File > > System.) Famfs supports formatting shared memory as a file system that can > > be simultaneously mounted from multiple hosts. > > > > Putting zero-copy data frames in famfs files allows jobs across a cluster > > to memory map data frames from a single copy in shared memory. This has the > > potential to deduplicate memory while reducing or avoiding sharding and > > shuffling overheads. > > > > Famfs files can be memory-mapped and used without awareness that the files > > are “special” (though creating famfs files does require special steps). > > Memory mapping a famfs file provides direct access to the memory – with no > > copying through the page cache. > > > > Famfs was published in February as a Linux kernel patch > > < > > https://lore.kernel.org/linux-fsdevel/ze5edu3jblefw...@dread.disaster.area/T/#m27639915e97443186b3ade9d1e94423bc58e6e22 > > > > > and a user space CLI and library; all are available on github > > <https://github.com/cxl-micron-reskit/famfs/blob/master/README.md>. The > > kernel patch set has been received seriously; if legitimate use cases are > > demonstrated, we expect it will make its way into mainline Linux – and we > > intend to step up and maintain it. > > > > Famfs is already usable with shared disaggregated memory (though this > > memory is not commercially available yet). Conventional memory can be > > shared among virtual machines today, to build (admittedly scaled down) > > POCs. > > > > I am looking for the following feedback: > > > > - Any questions are welcome, on or off-list. > > - Please tell us what sorts of work flows you might try with famfs > > shared memory if you had it – we are looking for ways to demonstrate use > > cases. > > - Help us get the word out. Are there people, groups, forums or > > conferences where we should introduce this capability? > > - If you are interested in testing famfs, please do – and let me know > > how we can help. > > > > Micron’s interest is in enabling an ecosystem where shared memory is > > practically usable. If famfs is successful, other access methods will > > surely emerge. Famfs is our attempt to enable shared memory via an > > existing, ubiquitous interface – making it easy to use without having to > > adopt new abstractions in advance. > > > > Thanks for reading, > > John Groves > > Micron > >
