HI William, You have to differentiate the server side registration from the client side registration. The links above are server side. There we allocate memory in larger segments (defined by the config variable crail.allocationSize). AllocationSize must be a multiples of crail.bufferSize which is the basic allocatoin unit for files. Reason we choose to allocate larger segments is to keep the state on the NIC small (each registration consumes state), and to amortize the regMr calls, but also to minimize the number of messages between the storage server and the metadata server (a storage server needs to inform the metadata server about each allocated segment).
On the client side we allocate memory that is used for buffered streams. This memory is also allocated in larger segments for the same reason to keep the registration state on the NIC small and to amortize the overhead of regMr calls. We further try to use huge pages for both client and server side which further reduces the state on the NIC. Let me know if you need further information. -Patrick On Mon, Mar 25, 2019 at 12:18 AM [email protected] <[email protected]> wrote: > > Hello, > > I am learning Crail and RDMA implementation. I have a question regarding > RDMA + Crail and I was hoping you could help me figure it out. In RDMA, a > memory space is required to be registered with a protection domain to get a > memory region (IbvMr) and respected rkey. However, each registration will > take extra overhead. > > After studying Crail code, I found it seems Crail only registers memory > space once with a single protection domain to get a single memory region > and save registration overhead, and all incoming connections/storageClients > share the same memory space, memory region, and protection domain with the > same RDMA privilege. Did I understand this correctly? > > Reference code in Crail: > 1 [create endpoint]. > https://github.com/apache/incubator-crail/blob/v1.1/storage-rdma/src/main/java/org/apache/crail/storage/rdma/RdmaStorageServer.java#L68 > 2 [register memory]. > https://github.com/apache/incubator-crail/blob/v1.1/storage-rdma/src/main/java/org/apache/crail/storage/rdma/RdmaStorageServer.java#L112 > > William >
