On 26 Oct 2012, at 23:40, Jeffrey Altman wrote: > I have significant concerns about the design of TCP OOB as it was > described at EAKC2012.
On the contrary, I think Out of Band support for AFS is a very interesting prospect. As I discussed with Andrew and Hartmut in Edinburgh, I think we could use an OOB negotiation protocol to support a lot of interesting innovations, from TCP transfers, to /vicep-access style use of an underlying cluster filesystem, to cluster specific data transfer protocols. The problem is that RX is trying to solve two discrete problems. On the one hand it carries metadata RPCs. These are very short lived transfers, typically comprised of a single packet in each direction. For AFS to appear fast to the end user (the latency problem CERN discussed in their presentation) we need to be able to handle these single packets as quickly as possible. On the other hand, RX also carries AFS's bulk data. Bulk data transfers tend to be large packet flows, and any negotiation overhead is quickly mitigated if it makes the transfer faster. Being able to use a different transport for bulk operations actually makes a lot of sense. Other filesystems, such as Lustre, have already gone down this path. I do believe that we can make the current RX implementation significantly faster - and that this will aid both bulk and metadata operations. However, it is unlikely that we can ever reach the raw performance of TCP, especially when TCP is aided by in-kernel DMA splitting packet decoding across multiple cores, and by specialised firmware in the network cards themselves. Whilst there is increasing evidence that UDP based protocols are more efficient than TCP ones in some network scenarios, they gain this added efficiency by using different flow control models than TCP. Whilst RX has a solely TCP inspired design, we're not going to surpass TCP. One really interesting possibility of generic out of band support is that we could support transports such as UDT, when we know that we're on a network that can deliver that kind of performance. So, I'm firmly in favour of working on standardising a mechanism for negotiating out of band transfers. I think it's also worth bearing in mind that the implementation that Andrew outlined was a proof-of-concept version. One issue that we, as a community, are going to have to decide is to what extent we're prepared to keep adding new features that only work on a single operating system. I would also really like to see some performance numbers for its use from multiple clients on a loaded fileserver. But we can cross those bridges as part of the development process. Cheers, Simon_______________________________________________ OpenAFS-info mailing list [email protected] https://lists.openafs.org/mailman/listinfo/openafs-info
