Uninstall winOFED Reboot Run ibcleanup.bat Reboot Run ibscan; re-run ibcleanup if necessary. Reboot Now try your unattended install.
>-----Original Message----- >From: Matthew Wallis [mailto:[email protected]] >Sent: Wednesday, April 20, 2011 2:50 PM >To: Smith, Stan >Cc: [email protected] >Subject: Re: [ofw] OFED for Windows on 2008R2 SP1 > > >Hi Stan, > Meant to get back to this earlier, I have been able to confirm > that the IPoIB driver is >the OpenFabrics one. I have run the ibscan.bat and had a look at what that >shows me, haven't used the >IBCleanup as yet. > >Matt. > >On 16/04/2011, at 2:42 AM, Smith, Stan wrote: > >>> -----Original Message----- >>> From: Matthew Wallis [mailto:[email protected]] >>> Sent: Friday, April 15, 2011 2:17 AM >>> To: Smith, Stan >>> Cc: [email protected] >>> Subject: Re: [ofw] OFED for Windows on 2008R2 SP1 >>> >>> >>> Apparently we've found a solution for this. Of a sort. >>> >>> The problem appears to be that SP1 comes with a driver for the Mellanox >>> card. If we remove that >>> driver, the machines actually stop booting. Nice. >> >> I suspect you have not entirely removed the drivers; 3 .sys files: >> mlx4_bus.sys, mlx4_hca.sys and >ipoib.sys. >> Try the ibscan/ibcleanup mentioned in previous email prior to install. >> >>> >>> Our current solution that obviously will require some more investigation is >>> to install the official >>> Mellanox drivers over the top of the Microsoft ones, then removing the >>> Mellanox drivers. Once those >>> are removed, the OFED drivers install without issue. >>> >>> That's the long way round. >>> >>> I can blacklist the driver in WSUS but I'm not sure I can stop SP1 from >>> installing it. >> >> I do not recollect SP1 installing the HCA drivers proper, although Windows >> update did want to >install the HCA drivers after the SP1 install/reboot. >> BTW, the MS update drivers are Mellanox drivers which have been WHQL'ed; MS >> does not distribute MS >developed HCA drivers at this time. >> >>> >>> Matt. >>> >>> On 15/04/2011, at 9:52 AM, Matthew Wallis wrote: >>> >>>> <ua-install.log><Screen shot 2011-04-15 at 9.32.59 AM.png> >>>> >>>> Using the GUI, yes, IPoIB works fine. We've got two nodes running on GPFS >>>> from manual installs. >> >> I'm curious as to 'which' vendor's IPoIB you are running? >> In windows\system32\drivers\ipoib.sys hold the mouse point over the filename >> and see what pops up? >> You should see a company name 'OpenFabrics Alliance'. >> >> >>> Looks like it's a MT26428 >>>> These are IBM branded cards with their own firmware on them. The Mellanox >>>> OEM and general drivers >>> don't load at all on >>>> these cards, we've had to push IBM to say they will support the OFED >>>> drivers. >> >> >> You are in uncharted waters w.r.t. these IBM branded HCAs. >> To the best of my understanding, winOFED HCA drivers (mlx4_bus/mlx4_hca) >> have never been loaded on >IBM hardware. >> Good to understand the GUI install worked. >> >> Stan. >> >> >>>> >>>> [root@m1-m ~]# ibstat >>>> CA 'mlx4_0' >>>> CA type: MT26428 >>>> Number of ports: 1 >>>> Firmware version: 2.7.700 >>>> Hardware version: b0 >>>> Node GUID: 0x0002c903000d8d02 >>>> System image GUID: 0x0002c903000d8d05 >>>> Port 1: >>>> State: Active >>>> Physical state: LinkUp >>>> Rate: 40 >>>> Base lid: 1 >>>> LMC: 0 >>>> SM lid: 1 >>>> Capability mask: 0x0251086a >>>> Port GUID: 0x0002c903000d8d03 >>>> Link layer: IB >>>> >>>> >>>> <Screen shot 2011-04-15 at 9.47.27 AM.png> >>>> >>>> On 15/04/2011, at 1:49 AM, Smith, Stan wrote: >>>> >>>>> Which ConnectX HCA and which firmware version? >>>>> >>>>> Mellanox website has firmware burning tools if not latest firmware. >>>>> >>>>> Please try the following install cmd: start/wait msiexec /i >>>>> OFED_2-3_win7_x64.msi /qn /log >c:\ua- >>> install.log >>>>> >>>>> forward ua-install.log >>>>> >>>>> Thanks, >>>>> >>>>> Stan. >>>>> >>>>>> -----Original Message----- >>>>>> From: Matthew Wallis [mailto:[email protected]] >>>>>> Sent: Wednesday, April 13, 2011 9:05 PM >>>>>> To: Smith, Stan >>>>>> Cc: [email protected] >>>>>> Subject: Re: [ofw] OFED for Windows on 2008R2 SP1 >>>>>> >>>>>> >>>>>> Just did the certificate based install using the msi, same results, only >>>>>> no prompting for certs >>> this >>>>>> time >>>>> >>>>> Progress. >>>>> >>>>>> >>>>>> z:\support\OFED> rem-cert-add.bat OFED OpenFabrics.cer >>>>>> Installing OpenFabrics.cer Cert on M1031 >>>>>> >>>>>> M1031 SUCCESS: OFA TrustedPublisher cert installed >>>>>> >>>>>> z:\support\OFED>start/wait msiexec /i OFED_2-3_win7_x64.msi /qn >>>>>> >>>>>> z:\support\OFED> >>>>>> >>>>>> Had the usual flash message about the mlx_bus, but no IPoIB drivers, and >>>>>> vstat still complains >>> about >>>>>> missing complib.dll >>>>> >>>>> Since mlx4_bus does not appear to load correctly, then the mlx_hca driver >>>>> will not be loaded, >hence >>> no complib.dll and vstat failure. >>>>> In the past, HCA firmware problems have caused this type of behavior; >>>>> although no current idea as >>> to why a GUI install would work while the unattended install fails? >>>>> >>>>> Speaking of a GUI install, does IPoIB configure correctly? >>>>> >>>>>> >>>>>> Matt. >>>>>> >>>>>> On 14/04/2011, at 10:22 AM, Matthew Wallis wrote: >>>>>> >>>>>>> We're doing an unattended install of Server 2008R2 HPC edition from a >>>>>>> network share, so I >>>>>>> can either try to pre-seed that with the OFED drivers, or I can try and >>>>>>> install the MSI via the >>> HPC >>>>>>> admin console. I'll give them a shot. I do need to make sure I get >>>>>>> IPoIB as we're using GPFS >for >>>>>>> the filesystem. >>>>>>> >>>>>> >>>>>> -- >>>>>> Matthew Wallis, HPC Systems Administrator >>>>>> Victorian Partnership for Advanced Computing. >>>>>> Ph: +61 3 9925 4452 Fax: +61 3 9925 4647 >>>>> >>>> >>>> -- >>>> Matthew Wallis, HPC Systems Administrator >>>> Victorian Partnership for Advanced Computing. >>>> Ph: +61 3 9925 4452 Fax: +61 3 9925 4647 >>>> >>>> _______________________________________________ >>>> ofw mailing list >>>> [email protected] >>>> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ofw >> _______________________________________________ ofw mailing list [email protected] http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ofw
