Attendees:
Leonid Killer (Mellanox)
Sean Hefty (Intel)
Eric Lantz (Microsoft); thank you Eric for sending in meeting notes!
Wu Wenhao (Microsoft)
Network Direct V2:
Specification published
V2 provider SW will not be ready for winOFED 3.0 release (perhaps a future
point release).
Mellanox ConnectX-3 FDR support:
o Ready in late August
o Relatively few source changes
Leonid believes a code commit into OpenIB SVN tree by August 31st is possible
due to minimal changes to mlx4 driver.
Leonid agreed to producing an SVN branch containing FDR code which interested
parties can review.
Given the small number of changes, perhaps submitting patches to the ofw list
for review would be faster for all?
ROCE support:
o Changes in many components of the stack
· What is nature of the changes?
Sending individual patches for review will be too time consuming due to extent
of changes.
mlx4_bus\* is an entire replacement
ibat extended to support ROCE; IPoIB changes?
Ibal mods
· No way to accept only part of the changes though
§ Lots of changes in the bus driver
o How to commit changes to svn?
· One big change in end of Aug
· Or many small changes which is hard to ETA
· Stan: Perhaps make a branch for the MLX4 folder structure and content?
Leonid agrees.
§ Then do inspection and bug hunting on that branch
Leonid agreed to create an SVN branch with ROCE specific changes; with so many
changes creating patch files is very time consuming.
Stan requested the ROCE branch be based on the latest OpenIB SVN trunk so ROCE
changes in the branch are easily identifiable.
Interested WWG parties will review branch ASAP.
Leonid noted that with so many changes for ROCE it will be difficult to review
and commit into the OpenIB SVN tree by the proposed code freeze date of Aug
31st.
Stan suggested the proposed winOFED 3.0 code freeze date be pushed out to
September 15th.
Leonid thought that ROCE commits might be able to be finished by 9/15 although
he needed to talk with the rest of the Mellanox Windows team.
Mellanox (binary only) Ethernet driver required for ROCE support:
o Including binary driver is contrary to philosophy of OSS
o In the past, ND provider was taken as a binary as a temporary measure only
with commitment from MS to check in the ND source when the MS-OFA legal process
on contributions was completed.
o Stan: Suggest Mellanox make the Ethernet binary available at their website
but not in winOFED.
· User would have to download from 2 locations to use WinOFED ROCE
· But that driver requires a totally different MELL bus driver when
using Ethernet
§ The driver depends on Mellanox bus driver AND the IB stack (including IBAL)
for ROCE
o Leonid would like to discuss in email thread with Uri & others to which
Stan agrees
Due to Intellectual Property disclosed in the Mellanox ROCE Ethernet driver
source for Windows, Mellanox proposed they supply a binary only driver to be
included in the winOFED 3.0 release.
After some offline discussions, all members of the WWG except Mellanox agreed
including a binary only driver in the winOFED 3.0 release was an unacceptable
solution as binary only inclusion violated the intent of the OFA open-source
project. The issue being open-source binaries are re-creatable from source
located in the open-source project source repository.
Leonid correctly pointed out that a binary only inclusion had been done before
with the Microsoft ND provider.
Eric Lantz pointed out the Microsoft ND provider binary was included only
because MS legal and OFA legal were still in process of resolving legal wording
and Microsoft agreed upfront to SVN commit the ND provider sources as soon as
MS legal would let them; the ND provider sources were svn committed shortly
after the legal issues were resolved.
Stan further pointed out how disruptive binary inclusion was to the winOFED
build process (numerous temporary modifications to the build process including
manual hands-on steps).
Mellanox had previously stated the Mellanox Ethernet driver sources would not
be svn committed due to exposure of IP in the source.
Stan noted the same IP issues were present in the Linux OFED ROCE Ethernet
driver sources, so why is the Windows driver different?
None the less, the WWG agreed a binary only ROCE Ethernet driver solution was
unacceptable.
Leonid asked if winOFED wanted ROCE support, Stan replied yes although the WWG
is willing to live without ROCE support if there is no alternative beyond a
binary only ROCE Ethernet driver solution.
Stan further suggested Mellanox could publish their Windows ROCE Ethernet
driver on the Mellanox website; similar to how Mellanox HCA firmware tools are
published.
When winOFED customers wish to use ROCE hardware over Ethernet, they would
install winOFED and then download/install the Mellanox Ethernet driver; (a
win-win story).
Stan suggested it's in Mellanox' s best interest to be able to update the
Mellanox Ethernet driver binaries and WHQL said Ethernet driver independent of
winOFED involvement.
Leonid then pointed out the ROCE mlx4_bus driver has link time dependencies on
the Mellanox Ethernet driver.
Everyone, outside of Mellanox, agreed a link time dependency on the Mellanox
Ethernet driver was unacceptable in that the winOFED stack would not load
without the Mellanox binary only Ethernet driver being present.
Sean requested more information about other link time dependencies the ROCE
code base had on other winOFED modules.
Stan proposed a modification to the ROCE mlx4_bus driver such that when the
Mellanox Ethernet driver is loaded it's Interface is exported to the ROCE
mlx4_bus driver thus removing the winOFED stack link time dependency on the
Mellanox ROCE Ethernet driver.
Leonid agreed to start an ofw email thread discussing how the link time
dependency on the Mellanox Ethernet driver could be removed and how the
mlx4_bus driver would then receive notification when the Mellanox ROCE Ethernet
driver interface was available. The interface notification mechanism would be
similar to how IBAL and Winverbs receive notification from HCA drivers today.
winOFED 3.0 release schedule:
o Stan proposed moving the WinOFED 3.0 code free to Sept15
· To get ROCE and FDR into v3.0
o Ready to start the RC1 test pass on Sept19
· Desirable to have hardware in place for ROCE and FDR testing in the MS
labs by this date
§ How do MS, Intel, ?others? Get access to FDR hardware in Sept?
§ ROCE does run on ConnectX-2 hardware
· Also verify the download of Mellanox Ethernet driver bits from their
website works with these WinOFED v3.0 bits as part of this testing
Opens:
· Eric Lantz asked the question of how FDR hardware testing was to be
accomplished in that MS and Intel did not have access to FDR capable hardware.
Perhaps Mellanox can allocate time on their for-customer-use clusters which
have FDR hardware installed?
· Eric further noted MS success on using cable plug converters going
from QSFP (40gb) IB connectors to 10gb Ethernet switch (SFP?) plugs.
· Eric also questioned as to how ROCE was to be tested. Stan pointed out
he had Connect-X2 hardware which could be cabled back-2-back to test ROCE
functionality.
_______________________________________________
ofw mailing list
[email protected]
http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ofw
