Hello Issac, Thank you very much for your response. I lowered the map_on_demand value to 16 and now it works fine.
However, I had once concern, whether lowering down this map_on_demand value would impact the performance of Lustre or not? Thanks again. Vipul -----Original Message----- From: [email protected] [mailto:[email protected]] Sent: 16 February 2010 21:29 To: Vipul Pandya Cc: [email protected] Subject: Re: [Lustre-discuss] Lustre-1.8.1.1 over o2ib gives Input/Output error while executing lctl ping On Mon, Feb 15, 2010 at 09:45:10PM -0800, Vipul Pandya wrote: > ...... > -> I tried to load the ko2iblnd module as you have suggested. But still > I am unable to do 'lctl ping'. I am getting the same error as shown > below. > #> modprobe ko2iblnd map_on_demand=64 Please lower it to "map_on_demand=32". > #> modprobe lnet > #> lctl ping 102.88.88....@o2ib > failed to ping 102.88.88....@o2ib: Input/output error > #> dmesg > Lustre: Listener bound to eth2:102.88.88.188:987:cxgb3_0 > Lustre: Register global MR array, MR size: 0xffffffff, array size: 2 > fmr_pool: Device cxgb3_0 does not support FMRs > LustreError: 4122:0:(o2iblnd.c:1393:kiblnd_create_fmr_pool()) Failed to > create FMR pool: -38 > Lustre: Added LNI 102.88.88....@o2ib [8/64/0/0] > LustreError: 2453:0:(o2iblnd.c:801:kiblnd_create_conn()) Can't create > QP: -12, send_wr: 520, recv_wr: 18 > Lustre: 2453:0:(o2iblnd_cb.c:1953:kiblnd_peer_connect_failed()) Deleting > messages for 102.88.88....@o2ib: connection faile rdma_create_qp() failed with -ENOMEM again, even with a much smaller send_wr (520 vs 2056). If lowering map_on_demand still couldn't fix it, you'd need to look into HCA driver/firmware as to why it failed to create the QP (if there's enough memory for it). Isaac _______________________________________________ Lustre-discuss mailing list [email protected] http://lists.lustre.org/mailman/listinfo/lustre-discuss
