On Mon, 2008-01-21 at 10:28 -0500, Shipman, Galen M. wrote: > We are seeing failures setting up a QP using rdma_create_qp. > > This only occurs when: > init_qp_attr.cap.max_send_wr > init_qp_attr.cap.max_recv_wr > > Totals to more than 16K. > > I have queried the device attributes and I have found: > > ib_device_attr.max_qp_wr = 65535
-12 is -ENOMEM You may want to add some printk's to mthca_alloc_wqe_buf() in mthca_qp.c. I think you're failing in the line qp->wrid = kmalloc((qp->rq.max + qp->sq.max) * sizeof (u64), GFP_KERNEL); rq.max gets set the max_recv_wr, sq.max gets max_send_wr, so you're trying to allocate 128KB when those total 16K -- that's trying to allocate 32 contiguous pages, which I believe is the max RHEL4's kernel will let you do via kmalloc(). New kernels may have alleviated this somewhat -- my Fedora 8 box has a 1MB slab/slub cache, but good luck actually allocating that if the box has been up any length of time. I'm not sure why it would work under userspace, but I've not looked very hard either. Perhaps the IOMMU is coming into play there? -- Dave Dillow National Center for Computational Science Oak Ridge National Laboratory (865) 241-6602 office _______________________________________________ general mailing list [email protected] http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
