Hi Stan,
Meant to get back to this earlier, I have been able to confirm
that the IPoIB driver is the OpenFabrics one. I have run the ibscan.bat and had
a look at what that shows me, haven't used the IBCleanup as yet.
Matt.
On 16/04/2011, at 2:42 AM, Smith, Stan wrote:
>> -----Original Message-----
>> From: Matthew Wallis [mailto:[email protected]]
>> Sent: Friday, April 15, 2011 2:17 AM
>> To: Smith, Stan
>> Cc: [email protected]
>> Subject: Re: [ofw] OFED for Windows on 2008R2 SP1
>>
>>
>> Apparently we've found a solution for this. Of a sort.
>>
>> The problem appears to be that SP1 comes with a driver for the Mellanox
>> card. If we remove that
>> driver, the machines actually stop booting. Nice.
>
> I suspect you have not entirely removed the drivers; 3 .sys files:
> mlx4_bus.sys, mlx4_hca.sys and ipoib.sys.
> Try the ibscan/ibcleanup mentioned in previous email prior to install.
>
>>
>> Our current solution that obviously will require some more investigation is
>> to install the official
>> Mellanox drivers over the top of the Microsoft ones, then removing the
>> Mellanox drivers. Once those
>> are removed, the OFED drivers install without issue.
>>
>> That's the long way round.
>>
>> I can blacklist the driver in WSUS but I'm not sure I can stop SP1 from
>> installing it.
>
> I do not recollect SP1 installing the HCA drivers proper, although Windows
> update did want to install the HCA drivers after the SP1 install/reboot.
> BTW, the MS update drivers are Mellanox drivers which have been WHQL'ed; MS
> does not distribute MS developed HCA drivers at this time.
>
>>
>> Matt.
>>
>> On 15/04/2011, at 9:52 AM, Matthew Wallis wrote:
>>
>>> <ua-install.log><Screen shot 2011-04-15 at 9.32.59 AM.png>
>>>
>>> Using the GUI, yes, IPoIB works fine. We've got two nodes running on GPFS
>>> from manual installs.
>
> I'm curious as to 'which' vendor's IPoIB you are running?
> In windows\system32\drivers\ipoib.sys hold the mouse point over the filename
> and see what pops up?
> You should see a company name 'OpenFabrics Alliance'.
>
>
>> Looks like it's a MT26428
>>> These are IBM branded cards with their own firmware on them. The Mellanox
>>> OEM and general drivers
>> don't load at all on
>>> these cards, we've had to push IBM to say they will support the OFED
>>> drivers.
>
>
> You are in uncharted waters w.r.t. these IBM branded HCAs.
> To the best of my understanding, winOFED HCA drivers (mlx4_bus/mlx4_hca) have
> never been loaded on IBM hardware.
> Good to understand the GUI install worked.
>
> Stan.
>
>
>>>
>>> [root@m1-m ~]# ibstat
>>> CA 'mlx4_0'
>>> CA type: MT26428
>>> Number of ports: 1
>>> Firmware version: 2.7.700
>>> Hardware version: b0
>>> Node GUID: 0x0002c903000d8d02
>>> System image GUID: 0x0002c903000d8d05
>>> Port 1:
>>> State: Active
>>> Physical state: LinkUp
>>> Rate: 40
>>> Base lid: 1
>>> LMC: 0
>>> SM lid: 1
>>> Capability mask: 0x0251086a
>>> Port GUID: 0x0002c903000d8d03
>>> Link layer: IB
>>>
>>>
>>> <Screen shot 2011-04-15 at 9.47.27 AM.png>
>>>
>>> On 15/04/2011, at 1:49 AM, Smith, Stan wrote:
>>>
>>>> Which ConnectX HCA and which firmware version?
>>>>
>>>> Mellanox website has firmware burning tools if not latest firmware.
>>>>
>>>> Please try the following install cmd: start/wait msiexec /i
>>>> OFED_2-3_win7_x64.msi /qn /log c:\ua-
>> install.log
>>>>
>>>> forward ua-install.log
>>>>
>>>> Thanks,
>>>>
>>>> Stan.
>>>>
>>>>> -----Original Message-----
>>>>> From: Matthew Wallis [mailto:[email protected]]
>>>>> Sent: Wednesday, April 13, 2011 9:05 PM
>>>>> To: Smith, Stan
>>>>> Cc: [email protected]
>>>>> Subject: Re: [ofw] OFED for Windows on 2008R2 SP1
>>>>>
>>>>>
>>>>> Just did the certificate based install using the msi, same results, only
>>>>> no prompting for certs
>> this
>>>>> time
>>>>
>>>> Progress.
>>>>
>>>>>
>>>>> z:\support\OFED> rem-cert-add.bat OFED OpenFabrics.cer
>>>>> Installing OpenFabrics.cer Cert on M1031
>>>>>
>>>>> M1031 SUCCESS: OFA TrustedPublisher cert installed
>>>>>
>>>>> z:\support\OFED>start/wait msiexec /i OFED_2-3_win7_x64.msi /qn
>>>>>
>>>>> z:\support\OFED>
>>>>>
>>>>> Had the usual flash message about the mlx_bus, but no IPoIB drivers, and
>>>>> vstat still complains
>> about
>>>>> missing complib.dll
>>>>
>>>> Since mlx4_bus does not appear to load correctly, then the mlx_hca driver
>>>> will not be loaded, hence
>> no complib.dll and vstat failure.
>>>> In the past, HCA firmware problems have caused this type of behavior;
>>>> although no current idea as
>> to why a GUI install would work while the unattended install fails?
>>>>
>>>> Speaking of a GUI install, does IPoIB configure correctly?
>>>>
>>>>>
>>>>> Matt.
>>>>>
>>>>> On 14/04/2011, at 10:22 AM, Matthew Wallis wrote:
>>>>>
>>>>>> We're doing an unattended install of Server 2008R2 HPC edition from a
>>>>>> network share, so I
>>>>>> can either try to pre-seed that with the OFED drivers, or I can try and
>>>>>> install the MSI via the
>> HPC
>>>>>> admin console. I'll give them a shot. I do need to make sure I get IPoIB
>>>>>> as we're using GPFS for
>>>>>> the filesystem.
>>>>>>
>>>>>
>>>>> --
>>>>> Matthew Wallis, HPC Systems Administrator
>>>>> Victorian Partnership for Advanced Computing.
>>>>> Ph: +61 3 9925 4452 Fax: +61 3 9925 4647
>>>>
>>>
>>> --
>>> Matthew Wallis, HPC Systems Administrator
>>> Victorian Partnership for Advanced Computing.
>>> Ph: +61 3 9925 4452 Fax: +61 3 9925 4647
>>>
>>> _______________________________________________
>>> ofw mailing list
>>> [email protected]
>>> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ofw
>
_______________________________________________
ofw mailing list
[email protected]
http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ofw
