[openib-general] comments on DAT registry in OpenIB

[Kanevsky, Arkady]

Title: Message

Dear DAT and OpenIB members,

There is a debate going on on OpenIB and DAT reflectors which is going around about kDAT registry for Linux.

 

I would like to review the requirements we had agreed at DAT collaborative and captured in the kDAT

and uDAT specs and review DAT registry in OpenIB from that prospective.

 

 

I would like to make it clear that I do NOT speak on behalf of DAT Collaborative but

as just one of its members.

 

1. Ability of Consumers to open IA based on its name
   • OpenIB supports it
2. Support for Consumers to get a list of available IAs to open
• OpenIB kDAT and uDAT registry provide this
• OpenIB kDAT registry no longer provide Provider attributes as stated above
• Preserves dat_registry_list_providers
• for kDAPL OpenIB changed dat_provider_info format so binary compatibility not preserved, but source compatibility is preserved.
• dat_provider_info differs between uDAPL and kDAPL in OpenIB
3. Ability to enumerate available IAs and their attributes
• OpenIB supports that for uDAPL unchanged from DAPL SF RI
• for kDAPL openIB supports a single type of thread-safety defined by the Linux kernel and the version of Linux kernel defines the kDAPL APIs that kernel version supports.
• for kDAPL query will not return DAT version and thread safety Provider attribute.
4. Map IA_name to Provider library (kDAPL or uDAPL)
   • OpenIB kDAPL and uDAPL support this
5. Ability for DAT providers to dynamically register and deregister DAPL Provider
• OpenIB supports that for kDAPL and uDAPL
• All existing registry APIs at DAPL SF RI are preserved
6. Single static DAT registry - platform specific
   • kDAT and uDAT specs explicitly state that the DAT registry is defined by the platform and DAT collaborative provided an example of Registry for Linux and Windows and agree that the DAT provided registry should be used by all providers. This ensures that DAT Registry will support all Providers and DAT registry from one vendor does not block other providers.
   • OpenIB kDAT registry is the Linux platform DAT registry which achieves the goal of supporting all kDAPL providers. It also provides additional benefit that it is Linux core which maintain kDAT registry  instead of DAT Collaborative
   • OpenIB uDAT registry remains the DAT collaborative one unchanged.
   • We can discuss whether or not we want to get uDAT registry closer to the OpenIB kDAT one
   • The DAT registry for kDAPL and uDAPL are different at DAPL SF RI and OpenIB maintains it.
   • Some changes may be needed for kDAPL Registry hot plug support for OpenIB. How it may impact uDAPL registry.
7. ia_name is under system admin control
   • remains the same following a platform convention
8. IA can represent
   1. single port
   2. several ports
   3. several HBAs or RNICs
   4. multiple IAs represent the same port
   • OpenIB kDAPL currently implements #1. Members can submit code patches to support other choices 
   • OpenIB uDAPL remains the same with current implementation providing #1 under Provider control.
9. Support for Consumers to get a list of available IAs to open
   • OpenIB kDAT and uDAT registry provide this
   • OpenIB kDAT registry no longer provide Provider attributes as stated above
10. DAT registry supports loading multiple DAPL Providers into the same address space.
    1. A Provider library loaded into an address space once
    2. A Provider library unloaded only when all open instances of its IAs are closed
    3. The same Provider library can be loaded into multiple address spaces
    • OpenIB uDAPL continues to provide it
    • OpenIB kDAPL supports it
11. DAT registry shall support polymorphism (Provider independency)
1. Consumer call DAT functions by the DAT handle independently from Provider is used
2. DAT registry provides redirection
3. dat_ia_open is Provider specific and sets up redirection table per address space per Provider
• first time open ensures that table redirection for a Provider is set up
• OpenIB kDAT and uDAT registry provide that
• OpenIB kDAT registry preserves the DAT redirection table as defined by DAT Collaborative
• OpenIB kDAT registry preserves DAT_provider structure
• need to file errata to DAT to move dat_ia_close after dat_ia_query to match DAPL SF RI and OpenIB one for kDAPL and uDAPL
12. The DAT_handle structure first field provides a pointer for redirection
• OpenIB kDAT and uDAT registry support this
• DAT registry interpret the DAT_handle as a pointer to redirection table
13. DAT registry supports multiple dat_ia_open for the same library from the same and different address spaces
• OpenIB DAT registry preserves that
• OpenIB DAT registry supports refcount so the library is not closed until the last Consumer leaves and library is loaded into an address space once.

 So for uDAT Consumer and uDAT Provider which are compliant with uDAT spec continue to work as before from DAPL SF RI.

 

kDAT Consumer no longer can choose Provider by DAT version # and thread safety. The Linux kernel version defines which version of DAT and its thread safety. Since kernel ULP is specific to the kernel configuration this is OK.

If kernel module was choosing DAT Provider based on these attributes it is no longer needed. The kernel module MUST match what kernel configuration, including version number, provides. If kernel ULP behaves differently based on version of DAT API and Provider thread safety this switch will have to be replaced by kernel configuration/version switch. This is "standard" for kernel ULP modules.

 

kDAT Provider will be effected. The dat_info structure is changed. It no longer has fields for DAT major and minor versions and no field for thread safety. DAT Provider must match kernel configuration/version DAT API and thread-safety.

 

Others: kDAT Providers will need to maintain a separate tree for Linux than for generic DAT or DAT for other platforms with appropriate Linux kernel versioning/configuration support. For IB Providers it is not needed since OpenIB kDAT provides that.  For iWARP vendors if they do not plug-in into gen2 as HW driver then they need to match the OpenIB/Linux kDAPL version DAT APIs and threading.

 

DAT Provider registration is different for uDAPL and kDAPL. This is true at DAPL SF RI. And it is still the case for OpenIB.

 

The biggest difference from DAT Spec Linux Registry example and OpenIB kDAT registry is a uDAPL/KDAPL registry shared configuration file that contains all available Providers information. But this is just an example. How Registry keeps the Provider info is up to a platform. Consumer can not rely on the back door access to the static DAT registry database. They should use the dat_registry_list_providers to get that info. OpenIB kDAT registry preserves that API with a format change for dat_info structure usable by Consumers.

 

uDAT and kDAT Providers code sharing is slightly diminished.

 

In summary, kDAT registry in OpenIB fulfils all DAT requirements.

There is a small change in one of the structure which impacts kDAPL Consumers but is consistent with the way Linux kernel ULP modules operation. The biggest change for DAT Collaborative is that this body no longer "the sole Provider of DAT Registry". I view that as good news since Linux maintainers are much better source of kDAPL and uDAPL registry then DAT Collaborative.

 

Arkady Kanevsky                       email: [EMAIL PROTECTED]

Network Appliance                     phone: 781-768-5395

375 Totten Pond Rd.                  Fax: 781-895-1195

Waltham, MA 02451-2010          central phone: 781-768-5300

 

 

 

 

Arkady Kanevsky                       email: [EMAIL PROTECTED]

Network Appliance                     phone: 781-768-5395

375 Totten Pond Rd.                  Fax: 781-895-1195

Waltham, MA 02451-2010          central phone: 781-768-5300

 

 

_______________________________________________
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general