> For client connections that would work, but it can't work for a server > that uses fork() after listen(). > > Would it be possible to extend the librdmacm and libibverbs to: > - handle fork() after getting RDMA_CM_EVENT_CONNECT_REQUEST > which means there have to be some functions to destroy contexts like > struct rdma_cm_id, but without affecting the real connection. > Similar to close() on an FD that is used shared between two processes. > - export/import an interprocess token attached to the hierarchy of objects > similar to gss_export_sec_context/gss_import_sec_context. > Or a function which exports the contexts into a FD (of a pipe), > which could also transfer open FDs to the other end. > Maybe this could pass the information through a rdma/ibverb event channel? > > Are there any other possible solutions?
It would be great if an entire RDMA connection could be referred to by an fd. Today that's not the case. Doing this requires an abstraction above verbs. Rsockets provides a similar abstraction, but it doesn't actually return an fd. Rsockets does support fork() to some degree. It does this by establishing a TCP connection. The RDMA connection is not actually setup until the first data transfer occurs. This handles the case where a server calls fork() after listen(). It works with the apps that I've tested, but still imposes some restrictions on how fork() is used. I hit into other issues besides fork() trying to support existing socket applications. Without having an fd, other calls require working around. This included dup2(), sendfile(), and fstat(), but I'm sure there are many others. It would be nice if there were a way to associate an RDMA connection with an fd and under ideal conditions allow data transfers to occur within the userspace context. But allow the communication to migrate to the kernel if certain calls are invoked. This would provide very good performance under most conditions, yet support legacy apps. - Sean
