> > 9p, like aoe, is a ping-pong protocol. each message requires an ack.
> > therefore, the transport layer doesn't need flow control.
>
> Therefore, it is also not able to utilise bandwidth
> effectively over longhaul links. As an example, US coast
> to coast round trip time latency is about 100ms. Now consider
> fcp. Each worker thread of fcp does 8K read/writes. Due to
> pingponging, the *most* a thread can xfer coast to coast is
> 80KBps (for 16 threads, 1.28MBps). It is actually much worse
> since each thread doesn't even overlap reads with writes.
i think you are conflating a ping-pong protocol with the
limitation of a single outstanding message. there is no
reason that one needs a 1:1 correspondence between threads
and outstanding messages. nor does the protocol inherently
prohibit 1 write from becoming multiple messages.
in the case of aoe, you can have up to 2^32 - 1 outstanding
packets at the same time to each target. thus the theoretical
maximum would be
80 * (2^31 - 1) kbps per target.
the aoe driver (in contrib) has a limit of 128 8k jumbo frames
outstanding per target. the aoe driver has a fixed number of
thread per target.
> Short of a sliding window that is as large as the capacity of
> the pipe, you can't expect to keep it full. As usual one has
> to trade off simplicity vs performance.
i don't think so. the only thing a sliding window gives you
is < 1 ack per message sent.
-erik
