Hi William, Not sure I completely understand the question but here is an to what I think you're asking. A storage server is always bound to one NIC, so one protection domain. You can start multiple storage servers per node, for instance one per NIC (or any number of storage servers you like) which would then involve different protection domains. Running multiple storage servers per node is a perfectly valid configuration, Crail storage servers are designed to be "micro-servers" and are meant to be deployed like this.
>From the moment the storage server registers a region with the metadata server, the region itself looses any meaning. Regions are broken into blocks, and blocks are assigned to files and different clients my read or write different files concurrently. -Patrick On Mon, Mar 25, 2019 at 7:53 PM West Lafayette L <[email protected]> wrote: > Hello Patrick, > > Thank you for your detailed explanation. My question actually focuses on > the storage server side. > > I am also aware that multiple memory registrations will take significant > memory space for the state on the RDMA NIC and it will also increase the > message exchange between the storage server and the metadata server. > > More specifically, my question is does Crail register a Crail region > (assume crail.regionsize = 1GB) with different RDMA memory regions (IbvMr) > for different clients? In Crail, a storage server will host connections to > multiple clients, and clients use these connections to issue RDMA requests > to get data from Crail storage server. Does Crail > 1. use different RDMA memory regions (IbvMr) for single Crail region? (put > each connection under different protection domains, register the single > Crail region by N times to get N different RDMA memory regions, and give > different connections different memory regions) > 2. or it's a kind of shared memory region idea? (put all connections under > the same protection domain, register the Crail region once, and share the > single RDMA memory region between different connections which can keep > state on the NIC small)? > Based on code study and discussion, I suppose it's close to 2. > > Again, many thanks. > > William > > On Mon, Mar 25, 2019 at 2:42 AM Patrick Stuedi <[email protected]> wrote: > > > HI William, > > > > You have to differentiate the server side registration from the client > side > > registration. The links above are server side. There we allocate memory > in > > larger segments (defined by the config variable crail.allocationSize). > > AllocationSize must be a multiples of crail.bufferSize which is the basic > > allocatoin unit for files. Reason we choose to allocate larger segments > is > > to keep the state on the NIC small (each registration consumes state), > and > > to amortize the regMr calls, but also to minimize the number of messages > > between the storage server and the metadata server (a storage server > needs > > to inform the metadata server about each allocated segment). > > > > On the client side we allocate memory that is used for buffered streams. > > This memory is also allocated in larger segments for the same reason to > > keep the registration state on the NIC small and to amortize the overhead > > of regMr calls. We further try to use huge pages for both client and > server > > side which further reduces the state on the NIC. > > > > Let me know if you need further information. > > > > -Patrick > > > > > > On Mon, Mar 25, 2019 at 12:18 AM [email protected] <[email protected]> > > wrote: > > > > > > > > Hello, > > > > > > I am learning Crail and RDMA implementation. I have a question > regarding > > > RDMA + Crail and I was hoping you could help me figure it out. In > RDMA, a > > > memory space is required to be registered with a protection domain to > > get a > > > memory region (IbvMr) and respected rkey. However, each registration > will > > > take extra overhead. > > > > > > After studying Crail code, I found it seems Crail only registers memory > > > space once with a single protection domain to get a single memory > region > > > and save registration overhead, and all incoming > > connections/storageClients > > > share the same memory space, memory region, and protection domain with > > the > > > same RDMA privilege. Did I understand this correctly? > > > > > > Reference code in Crail: > > > 1 [create endpoint]. > > > > > > https://github.com/apache/incubator-crail/blob/v1.1/storage-rdma/src/main/java/org/apache/crail/storage/rdma/RdmaStorageServer.java#L68 > > > 2 [register memory]. > > > > > > https://github.com/apache/incubator-crail/blob/v1.1/storage-rdma/src/main/java/org/apache/crail/storage/rdma/RdmaStorageServer.java#L112 > > > > > > William > > > > > >
