I think FSAL_PT interfaces via IPC with another process, though that IPC may be in an out of tree library.
You might want to look at a lighter IPC than TCP/UDP. If you are defining your own IPC, obviously you can make the API for that mirror the FSAL API. Ganesha uses a single multi-threaded process. It uses thread pools to distribute work (so threads are not dedicated to specific clients). There are rwlocks at the cache inode layer that do MOST of the protection necessary for FSAL objects (the FSAL may have lists or tables of objects that need their own protection, and there are some "file descriptor" things that aren't sufficiently protected by the cache inode locks). Matt makes some good suggestions on FSALs to start looking at, however there are some bits and pieces that are not well implemented by those FSALs: Range lock integration - Ganesha can manage range locks entirely internally, however, if you have other remote file system protocols or local processes, you need to push range locks down into the underlying filesystem. To do this fully, there needs to be some way to support multiple lock owners and asynchronous blocking locks. Share reservation integration - Ganesha again can manage these all by itself, but again, if you want integration with other remote file system protocols, you need to drive these into the underlying file system. If you care about NFS v3 clients mounting sub-directories under the export, you need to make sure lookup_path works as expected. If you have multiple file systems where inode numbers and/or handles can collide, you need to do some work to handle that. You probably need some kind of cache invalidation upcall mechanism. Then there are things like ACLs... For the above concerns, the most implemented FSAL is FSAL_GPFS. FSAL_GLUSTER is also well implemented. Actually, FSAL_GLUSTER may also do some IPC stuff. Frank > Well, the proxy fsal proxies to another NFS server over TCP or UDP, so you > might find that helpful. > > The Ceph fsal is a fairly minimal example of a fsal that delegates to a library > with a posix-like api. > > The problem of how to do IPC between your fsal and a remote using a > custom protocol is not one Ganesha solves per se. You could do a lot of > things, obviously. > > Matt > > ----- "Dirk Jagdmann" <d...@cubic.org> wrote: > > > Hello developers, > > > > I'm currently evaluating how to best integrate NFS-Ganesha with our > > storage product. My company is developing a user space Linux > > application to handle storage and we'd like to provide a NFS service > > to access files and directories. > > > > As a proof of concept we've currently used the Linux FUSE feature to > > expose our storage as a regular Linux VFS handled filesystem, and > > NFS-Ganesha can use this just fine. However using FUSE to pass data > > between 2 user space processes is probably not the best solution and > > also probably not the fastest solution. I'm therefore currently > > investigating how feasible it is to develop a custom FSAL, which will > > communicate directly with another user space process. I'm thinking > > about writing a simple "network protocol" which will forward the FSAL > > calls to our storage user space process. It could use a stream or > > datagram style "connection", which could be regular TCP/UDP, maybe > > unix domain sockets, maybe Linux netlink sockets, maybe even SysV > > shared memory (if that's even necessary for performance). > > > > I did read all the NFS-Ganesha Wiki pages and read the fsal_api.h > > header once, and have a couple of questions now: > > > > - does the NFS-Ganesha server use multiple processes or multiple > > threads? > > > > - if it does use multiple process/threads, how are multiple connected > > NFS clients multiplexed to the processes/threads? > > > > - if multiple threads are used, how are the FSAL objects allocated and > > used with regards to the threads? Is locking required "inside" the > > FSAL implementation? > > > > - did anybody write a FSAL which forwards the calls made from > > NFS-Ganesha to the FSAL objects to another process? I'm thinking about > > writing a FSAL with a generic protocol which different user space > > processes can connect to and provide the storage/files/directories. > > > > - looking at the current FSAL implementations, which is the simplest > > to get me started, but implements all the necessary features? I'm > > thinking about studying the implementation and using that as a > > template for my own FSAL. > > > > > > -- > > ---> Dirk Jagdmann > > ----> http://cubic.org/~doj > > -----> http://llg.cubic.org > > > > ---------------------------------------------------------------------- > > -------- Don't Limit Your Business. Reach for the Cloud. > > GigeNET's Cloud Solutions provide you with the tools and support that > > you need to offload your IT needs and focus on growing your business. > > Configured For All Businesses. Start Your Cloud Today. > > https://www.gigenetcloud.com/ > > _______________________________________________ > > Nfs-ganesha-devel mailing list > > Nfs-ganesha-devel@lists.sourceforge.net > > https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel > > -- > Matt Benjamin > CohortFS, LLC. > 315 West Huron Street, Suite 140A > Ann Arbor, Michigan 48103 > > http://cohortfs.com > > tel. 734-761-4689 > fax. 734-769-8938 > cel. 734-216-5309 > > ---------------------------------------------------------------------------- -- > Don't Limit Your Business. Reach for the Cloud. > GigeNET's Cloud Solutions provide you with the tools and support that you > need to offload your IT needs and focus on growing your business. > Configured For All Businesses. Start Your Cloud Today. > https://www.gigenetcloud.com/ > _______________________________________________ > Nfs-ganesha-devel mailing list > Nfs-ganesha-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel ------------------------------------------------------------------------------ Don't Limit Your Business. Reach for the Cloud. GigeNET's Cloud Solutions provide you with the tools and support that you need to offload your IT needs and focus on growing your business. Configured For All Businesses. Start Your Cloud Today. https://www.gigenetcloud.com/ _______________________________________________ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
