Hi Christian, >>>> Nor why "fgds" is the name, that's going to be hard to remember, does it >>>> stand for something? >>>> >>> "FGDS" stands for Fast GPUDirect Storage. GPUDirect Storage (GDS) is >>> NVIDIA's >>> technology enabling direct I/O between GPU memory and files on NVMe, >>> widely used in LLM workloads to bypass CPU overhead. >> >> That's nvidia's specific solution, but this works on other devices, >> right? Or just for that one platform? > > That was nvidia's specific and very hacky out of tree solution Exactly, I completely agree with your viewpoint. In fact, we previously conducted a deep analysis of NVIDIA's GDS implementation code, which was also one of the motivations for us to develop fgds.
Please allow me to introduce NVIDIA's GDS implementation briefly: NVIDIA introduced two kernel modules: one module called nvidia-fs, and the other module is a customized NVMe driver replacing the Linux kernel's default NVMe driver. nvidia-fs creates a device file for every NVIDIA GPU on the machine: /dev/nvidia-fs<gpu-id>. Applications call cuFileBufRegister to register the GPU memory buffer to perform I/O. cuFileBufRegister executes the following operations: calls nvidia-fs via ioctl. The implementation of this ioctl allocates a corresponding phony buffer of the same size in the host memory, and establishes a mapping between the CPU memory phony buffer and the GPU memory buffer. Then, the application calls cuFileRead/cuFileWrite to perform file I/O operations. Its implementation calls ioctl on nvidia-fs with NVFS_IOCTL_READ/NVFS_IOCTL_WRITE as parameters. Inside the ioctl implementation, it calls the common kernel interface filp->f_op->read_iter/write_iter() on the file on NVME. These interfaces can only take the CPU memory phony buffer address as a input, constructing read/write requests sent through the block layer to the customized NVMe driver. The customized NVMe driver intercepts the I/O operations, calls the nvidia-fs interface to query, replaces the phony buffer address with the GPU memory dma address, and performs DMA transfer between GPU memory and NVMe. As we can see, this implementation is indeed very hacky and non-elegant. Furthermore, as evaluated in this paper published in SC'25 [1], the phony buffer brings a considerable performance overhead. > which as far as I know is pretty much abandoned everywhere. However, despite the overhead of phony buffers, GDS performance is still significantly higher than transferring through CPU host memory (as shown in our performance benchmark tests [2]). Therefore, GDS is actually still widely used in the LLM ecosystem. For instance, model loading plugins used in inference engines like vLLM and SGLang—such as fastsafetensors and InstantTensor— both support acceleration via GDS [3,4], with fastsafetensors enabling GDS by default for model loading. Furthermore, LMCache, a plugin used for KV cache offloading in vLLM and SGLang, also supports GDS acceleration [5]. PyTorch itself also provides file access APIs based on GDS [6]. > > AMD came up with something similar, but all those approaches are so > fundamentally broken that we didn't even considered upstreaming it. >> And you are using this as a "bypass" for the normal accel subsystem, >> shouldn't this be part of that subsystem instead of a custom user/kernel >> api like you are creating here? > > As far as I know there is a patch set under review and even already partially > merged which enables exactly that functionality as general feature for > DMA-buf which is vendor independent and should at least in theory work with > all drivers. > > I'm really surprised that somebody is still working on the vendor specific > stuff. As you pointed out, every vendor has been inventing their own way and interfaces to support GDS, introducing custom kernel modules and proprietary UAPI interfaces, with varying performance that leaves developers heavily frustrated. Apologies for not making this clear enough in our commit messages, which understandably caused some confusion. We merely borrowed the name "GDS" to describe the functional purpose of fgds. In fact, we believe fgds offers four key advantages: (1) GPU platform independence; (2) POSIX/io_uring interface compatibility; (3) Higher performance than GDS; (4) Minimal kernel footprint and UAPI footprint Regarding (1), (2), and (3), please allow me to briefly explain the design mechanism of fgds: fgds turns a GPU memory buffer into a POSIX/io_uring-compatible user-space virtual address via three main steps: Step 1: Utilizing ZONE_DEVICE support, we remap the GPU memory exposed via PCIe BAR into struct pages using devm_memremap_pages(); Step 2: Utilizing dma-buf support, the GPU memory buffer is exported as a dma-buf file descriptor (fd). Using this fd as a bridge, we look up the corresponding DMA addresses for the GPU memory buffer inside the kernel; Step 3: Through mmap, we insert the struct pages corresponding to the GPU memory buffer into the userspace VMA, mapping their physical/DMA addresses directly. The virtual address returned by mmap can then be directly passed into standard POSIX or io_uring interfaces. As you can see, since almost all major GPU vendors support exporting GPU memory buffers via dma-buf, all remaining technical dependencies of fgds rely on standard Linux kernel infrastructure. Therefore, fgds is completely vendor-agnostic and natively compatible with POSIX/io_uring without introducing any proprietary vendor interfaces, which greatly simplifies development, deployment, operations and unifies standard usage. Furthermore, because this technique completely eliminates the phony buffer, its performance is significantly better than NVIDIA's GDS (as shown in our benchmarks [2]). In addition, since fgds uses only the most fundamental dma-buf mechanisms, it relies on baseline dma-buf features that have been supported in the upstream kernel for a long time, rather than any new dma-buf features currently under active development. In fact, before we recently ported fgds to the latest kernel tree, it was developed and ran on our internal 6.6 kernel. It has been running stably in our production clusters for over half a year across various hardware platforms (including NVIDIA, AMD, and several other vendors) without requiring a single line of GPU-platform-specific fgds code modification. Regarding (4): We believe that implementing GDS-like functionality inherently requires kernel assistance to map the GPU memory buffer to a valid userspace virtual address. This inevitably requires userspace-kernel interaction. To the best of our knowledge, the mainline kernel currently lacks a dedicated, unified path for this specific interaction, which is why various vendors ended up writing their own out-of-tree interfaces. In contrast, fgds introduces only one single new ioctl parameter (REG_BUFFER, excluding UNREG_BUFFER), and confines its scope strictly to a standalone device driver. We believe this achieves a minimal kernel footprint and minimal UAPI addition. [1] https://dl.acm.org/doi/10.1145/3712285.3759862 [2] https://github.com/Storage-and-OS-for-AI/fgds [3] https://github.com/foundation-model-stack/fastsafetensors/blob/main/docs/configuration.md [4] https://github.com/scitix/InstantTensor/blob/main/csrc/loader_io_cufile.cpp [5] https://github.com/LMCache/LMCache/blob/dev/lmcache/v1/storage_backend/gds_backend.py [6] https://docs.pytorch.org/docs/2.14/generated/torch.cuda.gds.GdsFile.html Thanks, Li Wang> Regards, > Christian. > >> >> thanks, >> >> greg k-h
