Tom Tucker wrote:
On Tue, 2008-05-27 at 12:39 -0400, Talpey, Thomas wrote:
At 11:33 AM 5/27/2008, Tom Tucker wrote:
So I think from an NFSRDMA coding perspective it's a wash...
Just to be clear, you're talking about the NFS/RDMA server. However, it's
pretty much a wash on the client, for different reasons.
Tom:
What client side memory registration strategy do you recommend if the
default on the server side is fastreg?
On the performance side we are limited by the min size of the
read/write-chunk element. If the client still gives the server a 4k
chunk, the performance benefit (fewer PDU on the wire) goes away.
Tom
I would hope that dma_mr usage will be replaced with fast_reg on both
the client and the server.
When posting the WR, We check the fastreg capabilities bit + transport
type bit:
If fastreg is true -->
Post FastReg
If iWARP (or with a cap bit read-with-inv-flag)
post rdma read w/ invalidate
... For iWARP's case, this means rdma-read-w-inv,
plus rdma-send-w-inv, etc...
Maybe I'm confused, but I don't understand this. iWARP RDMA Read requests
don't support remote invalidate. At least, the table in RFC5040 (p.22) doesn't:
-------+-----------+-------+------+-------+-----------+--------------
RDMA | Message | Tagged| STag | Queue | Invalidate| Message
Message| Type | Flag | and | Number| STag | Length
OpCode | | | TO | | | Communicated
| | | | | | between DDP
| | | | | | and RDMAP
-------+-----------+-------+------+-------+-----------+--------------
0000b | RDMA Write| 1 | Valid| N/A | N/A | Yes
| | | | | |
-------+-----------+-------+------+-------+-----------+--------------
0001b | RDMA Read | 0 | N/A | 1 | N/A | Yes
| Request | | | | |
-------+-----------+-------+------+-------+-----------+--------------
0010b | RDMA Read | 1 | Valid| N/A | N/A | Yes
| Response | | | | |
-------+-----------+-------+------+-------+-----------+--------------
0011b | Send | 0 | N/A | 0 | N/A | Yes
| | | | | |
-------+-----------+-------+------+-------+-----------+--------------
0100b | Send with | 0 | N/A | 0 | Valid | Yes
| Invalidate| | | | |
-------+-----------+-------+------+-------+-----------+--------------
0101b | Send with | 0 | N/A | 0 | N/A | Yes
| SE | | | | |
-------+-----------+-------+------+-------+-----------+--------------
0110b | Send with | 0 | N/A | 0 | Valid | Yes
| SE and | | | | |
| Invalidate| | | | |
-------+-----------+-------+------+-------+-----------+--------------
0111b | Terminate | 0 | N/A | 2 | N/A | Yes
| | | | | |
-------+-----------+-------+------+-------+-----------+--------------
1000b | |
to | Reserved | Not Specified
1111b | |
-------+-----------+-------------------------------------------------
I want to take this opportunity to also mention that the RPC/RDMA client-server
exchange does not support remote-invalidate currently. Because of the multiple
stags supported by the rpcrdma chunking header, and because the client needs
to verify that the stags were in fact invalidated, there is significant
overhead,
and the jury is out on that benefit. In fact, I suspect it's a lose at the
client.
Tom (Talpey).
_______________________________________________
general mailing list
[email protected]
http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
_______________________________________________
general mailing list
[email protected]
http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
_______________________________________________
general mailing list
[email protected]
http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
