This may be your problem: MaxReadReq 128 bytes U need a BIOS update , common value with good performance is 512. Alternatively u can force it using setpci but than system may be unstable.
-----Original Message----- From: Bart Van Assche [mailto:[EMAIL PROTECTED] Sent: Friday, January 18, 2008 10:56 AM To: Sagi Rotem Cc: Openib-General Subject: Re: [ofa-general] Performance of MT25204 versus MT25208 > On January 17, 2008 Bart Van Assche wrote: > > > > Via ib_rdma_bw and ib_rdma_lat I obtained the following performance > > data: > > * peak bandwith 674 MB/s / latency 3.07 microseconds between two > > uniprocessor systems with MT25204 HCA's. > > * peak bandwith 935 MB/s / latency 3.36 microseconds between two > > dual processor systems with MT25208 HCA's. > > > > I'm wondering now whether the performance differences are due to the > > HCA's or due to the motherboard ? > On Jan 17, 2008 5:13 PM, Sagi Rotem <[EMAIL PROTECTED]> wrote: > Did you check that the PCIe slot you are using is x8 The motherboard is an Intel(R) Server Board S5000PAL (http://www.intel.com/design/servers/boards/s5000pal/), which has both x4 and x8 slots. I'm not sure the information from lspci can be trusted, but lspci says it's a x8 slot: # lspci -vv ... 08:00.0 InfiniBand: Mellanox Technologies MT25204 [InfiniHost III Lx HCA] (rev a0) Subsystem: Mellanox Technologies MT25204 [InfiniHost III Lx HCA] Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- Latency: 0, Cache Line Size: 64 bytes Interrupt: pin A routed to IRQ 16 Region 0: Memory at b9300000 (64-bit, non-prefetchable) [size=1M] Region 2: Memory at b8000000 (64-bit, prefetchable) [size=8M] Capabilities: [40] Power Management version 2 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-) Status: D0 PME-Enable- DSel=0 DScale=0 PME- Capabilities: [48] Vital Product Data Capabilities: [90] Message Signalled Interrupts: Mask- 64bit+ Queue=0/5 Enable- Address: 0000000000000000 Data: 0000 Capabilities: [84] MSI-X: Enable- Mask- TabSize=32 Vector table: BAR=0 offset=00082000 PBA: BAR=0 offset=00082200 Capabilities: [60] Express Endpoint IRQ 0 Device: Supported: MaxPayload 128 bytes, PhantFunc 0, ExtTag- Device: Latency L0s <64ns, L1 unlimited Device: AtnBtn- AtnInd- PwrInd- Device: Errors: Correctable+ Non-Fatal+ Fatal+ Unsupported- Device: RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop- Device: MaxPayload 128 bytes, MaxReadReq 128 bytes Link: Supported Speed 2.5Gb/s, Width x8, ASPM L0s, Port 8 Link: Latency L0s unlimited, L1 unlimited Link: ASPM Disabled RCB 64 bytes CommClk- ExtSynch- Link: Speed 2.5Gb/s, Width x8 ... Bart. _______________________________________________ general mailing list [email protected] http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
