Kanevsky, Arkady wrote:
Arlin,
nice proposal, thanks.
I have one high level question and a few specific technical ones.
1. Why do you want to provide this functionality via extension instead
of part of new DAT spec, say 2.0?
This will allow Consumers to use all events, operations, and
Provider/IA functionality uniformly instead
of via 2 separate layers. This will also ensure that this basic
funcionality can be provided by all DAPL Provider
the same way on DAPL and DAT layers.
DAPL 2.0 is not done yet so we have time to incorporate that.
DAPL 2.0 already introduced new functionality which is easy to beef up
for your proposal.
See DAT_DTOS for example. DAT_EVENT is also modified to handle remote
invalidation so
a small addition for Immediate data and Atoimc ops is a sensible addition.
This should simplify proposal significantly. As you will not need to
introduce any new
EXT structures.
As mentioned on the con-call, there are two separate items to consider
while looking at the proposal. The first is the ability to extend DAT
for specific provider value-add and the second is to validate the need
for general atomic and immediate data functionality in the basic set of
API's for all providers. I included atomics and immediate data as
examples since it is specific to one provider (IB), it includes
operations that require new ops, events, and event data types, and it
also provides a working model to validate the extension model from
request to completion events. I would like to concentrate on getting
consensus on the extension proposal first if possible. Just try to think
of the actual operations as some opaque dat_ext_foobar_op().
In general, extension route was intended for RNIC|HCA providers to
expose HW capabilities beyond
IBTA, iWARP and VIA standards. The standard RDMA functionality is best
handle via spec addition.
DAT 2.0 does it for FMR, remote and local memory invalidation as well
as others.
True, but the extension route is not fully defined, documented, nor
implemented. This is what I would like to work on getting completed in
time for 2.0 if possible.
BTW: The existing implementation actually uses dapl_provider->extension
to store the hca_ptr but the specification states that it is reserved
for the providers private use (8.2.1 in DAPL1.2 spec). This is why I had
to defined another extension_func in the patch.
I had posted a complete list of changes/addition to DAT 2.0 about a
month ago.
But we had not discussed yet version change from 1.3 to 2.0 nor how
much backwards compatibility spec
will provide.
2. What is IMMED_EVENT? is it just immediate data without any payload one?
I suggest chnaging the name so it will not use "EVENT". Just call it
NO_PAYLOAD.
Do you want to support 2 different way to delivery immediate data?
One in event and one in data payload?
Why? I would think that just an event way will do.
This was modeled after the immediate data discussions on the DAT
reflector based on iWARP requirements.
http://groups.yahoo.com/group/dat-discussions/message/3285
3. I suggest beefing up DAT_DTO_COMPLETION_EVENT_DATA and DAT_DTOS
to convey which operation completed and return Immediate data if
complete operation had immediate data.
Since we already modified these 2 struct as part of DAT 2.0 change
lets add your proposal to the change.
This will allow Consumers to use single approach to deal with
completions, extension to the current one
but not a structural one. No need for DAT_EXTENSION_DATA,
DAT_EXT_EVENT_TYPE, DAT_EXT_OP
nor the whole mechanism for extended ops.
You still need extension types for the "other" value-add
operations/evnts that will not be accepted as standard and are vendor
specific.
I would like to defer the rest of the questions for now since they touch
on actual operations and not the extension mechanism. Although, I do
need to think about how to extend memory registration privledges. Any
suggestions?
4. What is the purpose of DAT_EXT_WRITE_CONFIRM_FLAG? Is it to expose
IB round trip semantic?
iWARP does not support immediate data. One can try to format payload
to pass immediate data.
Is that what you had in mind?
What is the semantic meaning of the completion with this flag set?
without flag set?
Are extended flags are additonal values for COMPLETION_FLAGS? 2.4.1
talks about extended flags
but where they are passed in is not defined.
DAT 2.0 extended them already for FMR barrier. I would prefer to
follow that route rather than creating a separate
extension completion flags.
5. Why do you need RECV_IMMED? If Immed data is delivered in event no
new Recv operation is needed.
If Consumer asks for immediate data in payload where in payload will
it be?
If this is needed for local match for remote RDMA_Write to handle
immediate data lets state so.
What happens for mismatch between local and remote op? That is recv
was posted for Send and RDMA_Write
"arrived"? Vice Versa?
6. I see extension for immediate data for rdma_write but not for send.
Is this deliberate? If we are going
to extend DAT semantic to support Immediate data we can as well
support the full IBTA/iWARP functionality for it.
7. Currently memory registration do not support access to LMR or RMR
by Atomic ops.
Do you propose to extend the meaning of current MEM_PRIV for LMR and
RMR to covers atomic accesses
or add new values to LMR_MEM_PRIV and RMR_MEM_PRIV for atomic
operation support?
8. Any alignment requirements for memory used for atomic ops?
9. Any correlation requirements for SRQ buffers to support recv with
immediate data?
Have a great holidays,
Arkady
Arkady Kanevsky email: [EMAIL PROTECTED]
<mailto:[EMAIL PROTECTED]>
Network Appliance Inc. phone: 781-768-5395
1601 Trapelo Rd. - Suite 16. Fax: 781-895-1195
Waltham, MA 02451 central phone: 781-768-5300
_______________________________________________
openib-general mailing list
[email protected]
http://openib.org/mailman/listinfo/openib-general
To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
