Sean Hefty wrote: > From user space, the call sequence does not change. The user calls > rdma_resolve_addr, rdma_resolve_route, rdma_connect, etc. It is up to the > librdmacm to perform the resolution. Today, the resolution request is simply > passed down to the kernel, which restricts how the resolution can be > performed.
good, fair-enough > I kept resolving the address and route separate. rdma_set_ib_path, which has > always existed btw, simply sets the route/path. The new call, > rdma_set_ib_dest, sets the address mapping. To use rdma_set_ib_dest, the user > must have called rdma_bind_addr first, which covers steps 1 & 2 that you > mentioned above. The rdma_bind_addr call can be done internally to the > librdmacm as part of the rdma_resolve_addr implementation. I understand that rdma_bind_address covers the local device and vlan resolutions, but I we should also --keep-- supporting also applications that use an explicit source address in rdma_resolve_addr or that don't do bind, provide src=NULL to resolve_addr and rely on the rdma-cm to use route lookup (as the rdma_resolve_addr man page indicates) for the device/vlan resolution. > If a user sets the wrong address mapping or route, they should only affect > themselves I wasn't sure to follow this comment, can you elaborate a bit more? > (FYI - I have not yet implemented the librdmacm to call rdma_bind_addr as part > of rdma_resolve_route on linux. I did not see an easy way to convert a > destination IP address to a source IP address. If anyone knows how, please > let > me know.) I assume you was referring rdma_resolve_addr, correct? there should be a way to do that from user space and if not, you can go down to the kernel, resolve the device/vlan and then call ACM to resolve the destination. It seems that you must resolve the dev/vlan for issuing the ACM ARP replacement... > >I would prefer to have a solution where the app flow isn't touched, > >something like the kernel rdma-cm to communicate with the user space ACM > >daemon to get address and route resolutions. Does such a design makes > >sense to you? > Long term, this is exactly the type of flow that I envision. I'd like to have > real data to show that the ACM implementation scales first, which is part of > my > problem. I do not have the ability to easily change kernel drivers on any > larger sized clusters. My approach is to allow user space to perform the > address and route resolution and pass the data to the kernel. This way, we > have > the freedom to test multiple solutions, until we can settle on what works. I am not sure to fully follow on the easily-change-kernel-drivers claim, isn't some change to the kernel rdma-cm being a must for the ACM + librdmacm solution to work? suppose you have a way to fully do the addr+route resolutions from user space, will the kernel rdma-cm state machine will be willing to issue rdma_create_id rdma_set_ib_path (you said this exists today?) rdma_create_qp rdma_connect ??? Or. -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
