This series introduces IBNBD/IBTRS kernel modules.
IBNBD (InfiniBand network block device) allows for an RDMA transfer of block IO
over InfiniBand network. The driver presents itself as a block device on client
side and transmits the block requests in a zero-copy fashion to the server-side
via InfiniBand. The server part of the driver converts the incoming buffers back
into BIOs and hands them down to the underlying block device. As soon as IO
responses come back from the drive, they are being transmitted back to the
client.
Hi Jack, Danil and Roman,
I met Danil and Roman last week at Vault, and I think you guys
are awesome, thanks a lot for open-sourcing your work! However,
I have a couple of issues here, some are related to the code and
some are fundamental actually.
- Is there room for this ibnbd? If we were to take every block driver
that was submitted without sufficient justification, it'd be very
hard to maintain. What advantage (if any) does this buy anyone over
existing rdma based protocols (srp, iser, nvmf)? I'm really (*really*)
not sold on this one...
- To me, the fundamental design that the client side owns a pool of
buffers that it issues writes too, seems inferior than the
one taken in iser/nvmf (immediate data). IMO, the ibnbd design has
scalability issues both in terms of server side resources, client
side contention and network congestion (on infiniband the latter is
less severe).
- I suggest that for your next post, you provide a real-life use-case
where each of the existing drivers can't suffice, and by can't
suffice I mean that it has a fundamental issue with it, not something
that requires a fix. With that our feedback can be much more concrete
and (at least on my behalf) more open to accept it.
- I'm not exactly sure why you would suggest that your implementation
supports only infiniband if you use rdma_cm for address resolution,
nor I understand why you emphasize feature (2) below, nor why even
in the presence of rdma_cm you have ibtrs_ib_path? (confused...)
iWARP needs a bit more attention if you don't use the new generic
interfaces though...
- I honestly do not understand why you need *19057* LOC to implement
a rdma based block driver. Thats almost larger than all of our
existing block drivers combined... First glance at the code provides
some explanations, (1) you have some strange code that has no business
in a block driver like ibtrs_malloc/ibtrs_zalloc (yikes) or
open-coding various existing logging routines, (2) you are for some
reason adding a second tag allocation scheme (why?), (3) you are open
coding a lot of stuff that we added to the stack in the past months...
(4) you seem to over-layer your code for reasons that I do not
really understand. And I didn't really look deep at all into the
code, just to get the feel of it, and it seems like it needs a lot
of work before it can even be considered upstream ready.
We design and implement this solution based on our need for Cloud Computing,
the key features are:
- High throughput and low latency due to:
1) Only two rdma messages per IO
Where exactly did you witnessed latency that was meaningful by having
another rdma message on the wire? That's only for writes, anyway, and
we have first data bursts for that..
2) Simplified client side server memory management
3) Eliminated SCSI sublayer
That's hardly an advantage given all we are losing without it...
...
Cheers,
Sagi.
