Linux kernel = 2.6.18-92.1.26.el5_lustre.1.6.7.2smp OS = Centos 5.2 We have a Sun Blade system with Sun IB products (switch= Sun part number X2821A-Z 36 port QDR switch) (hcas = Sun part number X4216A-Z dual port DDR PCI-E)
I can SOMETIMES run mvapich or openmpi over IB and it works, but generally I get a "CQ polling error". So I went back to the rdma tests and see some problems. We have installed OFED 1.4.1-4, and because I was having problems I upgraded the firmware on the HCAS: lspci | grep -i infin 0b:00.0 InfiniBand: Mellanox Technologies MT25418 [ConnectX IB DDR, PCIe 2.0 2.5GT/s] (rev a0) mstflint -d 0b:00.0 q Image type: ConnectX FW Version: 2.6.0 Device ID: 25418 Chip Revision: A0 Description: Node Port1 Port2 Sys image GUIDs: 0003ba000100d770 0003ba000100d771 0003ba000100d772 0003ba000100d773 MACs: 0003ba00d771 0003ba00d772 Board ID: (SUN0060000001) VSD: PSID: SUN0060000001 An rping from the client to server gives verbose client created cm_id 0x10ca7c70 cma_event type RDMA_CM_EVENT_ADDR_RESOLVED cma_id 0x10ca7c70 (parent) cma_event type RDMA_CM_EVENT_ROUTE_RESOLVED cma_id 0x10ca7c70 (parent) rdma_resolve_addr - rdma_resolve_route successful created pd 0x10caa3d0 created channel 0x10caa3f0 created cq 0x10caa410 created qp 0x10caa550 rping_setup_buffers called on cb 0x10ca5010 allocated & registered buffers... cq_thread started. cma_event type RDMA_CM_EVENT_ESTABLISHED cma_id 0x10ca7c70 (parent) ESTABLISHED rmda_connect successful RDMA addr 10caaa90 rkey 2002800 len 100 send completion cma_event type RDMA_CM_EVENT_DISCONNECTED cma_id 0x10ca7c70 (parent) client DISCONNECT EVENT... wait for RDMA_WRITE_ADV state 6 cq completion failed status 5 rping_free_buffers called on cb 0x10ca5010 destroy cm_id 0x10ca7c70 An ib_rdma_lat gives local address: LID 0x16 QPN 0x004f PSN 0x743778 RKey 0x002500 VAddr 0x00000007c72001 remote address: LID 0x01 QPN 0x004f PSN 0x6497a1 RKey 0x002500 VAddr 0x00000018780001 Conflicting CPU frequency values detected: 2336.000000 != 2003.000000 Latency typical: inf usec Latency best : inf usec Latency worst : inf usec Any ideas????? -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
