Re: [openib-general] Drop in performance on Mellanox MT25204 single port DDR HCA
Quoting r. Woodruff, Robert J [EMAIL PROTECTED]: Subject: Drop in performance on Mellanox MT25204 single port DDR HCA Hi Roland/Michael, One of my coworkers in Champaign is seeing a performance issue with the latest SVN driver and the OFED 1.1 Mellanox driver on certain platforms. On the older SVN somewhere around 7500 the Mellanox driver did not save and restore certain PCI registers before a reset. Somewhere around SVN 8000 a patch was added to save and restore these registers. However on our Alcolu platform this patch causes the MaxReadReq to be set to 128 bytes (rather than 512) which limits bandwith to 650MBytes/sec. If I remove the save/restore of these registers (attached patch), the bandwidth is back to where we would expect it 1250 Mbytes/sec. Is there some problem with this patch or do you think it is some BIOS issue in the platform ? This is a BIOS issue - it should set the MaxReadReq register for maximum performance and stability. As a work-around, you can use the setpci utility to modify MaxReadReq before loading the driver. Unfortunately, mthca has no way to know which values are legal and which will give the best performance, and previous behaviour was out of spec, reportedly causing stability issues on compliant platforms. I will look into adding this info in release notes. -- MST ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] Drop in performance on Mellanox MT25204 single port DDR HCA
At 02:43 PM 10/2/2006, Roland Dreier wrote: Robert Yes. 1250Mbytes/sec is what we expect. You say the 128 Robert value comes from the BIOS ? If so, we need to discuss this Robert with our BIOS team to find out why they limit it to 128, Robert perhaps it is a BIOS bug. Yes, I believe that the BIOS is the only place that would set that value. We know that resetting the device makes it go back to a different default value, and nothing in the kernel that I know of is going to set it down to 128. 128B is the default minimum from PCIe so likely some BIOS engineer took a conservative view and chose the defaults (go figure). Setting Max Read Request Size = 4096 is preferred on any implementation as it is basically free from a chipset perspective. The chipset will likely return in cache line quantities but there is some obvious optimizations to be achieved by issuing a single DMA Read Request. Mike ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] Drop in performance on Mellanox MT25204 single port DDR HCA
Does using the tune_pci=1 module option for ib_mthca bring the performance back up? The reason the driver was changed to work this way is that presumably the BIOS is setting the PCI configuration as it does for a reason. So you might want investigate why the BIOS sets MaxReadReq down to 128 in the first place. (removing the save/restore across reset lets the HCA pick a new default for all the settings, but may cause problems by getting rid of BIOS settings, which we assume were done for a reason). However tune_pci=1 will make the driver override this setting if you really know what you're doing. - R. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] Drop in performance on Mellanox MT25204 single port DDR HCA
Roland wrote, However tune_pci=1 will make the driver override this setting if you really know what you're doing. - R. Peter, can you give this a try ? I think you set this in /etc/modprobe.conf add the line, options mthca tune_pci=1 Also, we need to understand why the BIOS in your platform is setting it to 128 rather than 512. Is this an oversight in the BIOS or are they doing it for a reason. woody ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] Drop in performance on Mellanox MT25204 single port DDR HCA
Adding: Options ib_mthca tune_pci=1 Puts MaxReadReq = 4096. I get 1250MB/s bandwidth. -- Peter -Original Message- From: Woodruff, Robert J Sent: Monday, October 02, 2006 3:51 PM To: Roland Dreier; Hartman, Peter Cc: Michael S. Tsirkin; openib-general; EWG; Hartman, Peter Subject: RE: Drop in performance on Mellanox MT25204 single port DDR HCA Roland wrote, However tune_pci=1 will make the driver override this setting if you really know what you're doing. - R. Peter, can you give this a try ? I think you set this in /etc/modprobe.conf add the line, options mthca tune_pci=1 Also, we need to understand why the BIOS in your platform is setting it to 128 rather than 512. Is this an oversight in the BIOS or are they doing it for a reason. woody ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] Drop in performance on Mellanox MT25204 single port DDR HCA
Adding: Options ib_mthca tune_pci=1 Puts MaxReadReq = 4096. I get 1250MB/s bandwidth. Is that good? I lost track from the beginning of the thread. I would suggest working with your platform people to figure out why the BIOS is setting the PCI Express parameters to non-optimal values. - R. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] Drop in performance on Mellanox MT25204 single port DDR HCA
Roland wrote, Is that good? I lost track from the beginning of the thread. I would suggest working with your platform people to figure out why the BIOS is setting the PCI Express parameters to non-optimal values. - R. Yes. 1250Mbytes/sec is what we expect. You say the 128 value comes from the BIOS ? If so, we need to discuss this with our BIOS team to find out why they limit it to 128, perhaps it is a BIOS bug. woody ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] Drop in performance on Mellanox MT25204 single port DDR HCA
Robert Yes. 1250Mbytes/sec is what we expect. You say the 128 Robert value comes from the BIOS ? If so, we need to discuss this Robert with our BIOS team to find out why they limit it to 128, Robert perhaps it is a BIOS bug. Yes, I believe that the BIOS is the only place that would set that value. We know that resetting the device makes it go back to a different default value, and nothing in the kernel that I know of is going to set it down to 128. - R. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
