Pete,
>
> You have three IB cards in the client? Wow. You might take a look
> at the output of "ibv_devices" and "ibv_devinfo" to see if the first
> NIC is the one that is connected to your server. Please send me the
> output too. If not, we will have to add code to let you specify an
> interface name to PVFS.
>
> The error comes from this line (simplified):
>
> mr = ibv_reg_mr(pd, buf, len, IBV_ACCESS_LOCAL_WRITE |
> IBV_ACCESS_REMOTE_WRITE | IBV_ACCESS_REMOTE_READ);
>
> where len is 20 * 8 kB == 160 kB. That's a pretty small amount
> of memory. This is the first memory registration call on the
> client.
You where right about the memory registration limitation. That was the
problem. installed pvfs on an x86_64 client and ran pvfs2-cp again.
Same results but this time I got a more informative message:
hpcxe005(salmr0)46:/usr/local/opt/pvfs-2.6.3/bin/pvfs2-cp
-t /test/pvfs2/testfile1 /test/pvfs2/testfile2
libibverbs: Warning: RLIMIT_MEMLOCK is 32768 bytes.
This will severely limit memory registrations.
[E 09:24:11.223942] Error: openib_ib_initialize: ibv_create_cq failed.
[E 09:24:11.224338]
[bt] /usr/local/opt/pvfs-2.6.3/bin/pvfs2-cp(error+0xca) [0x456e1a]
[E 09:24:11.224373]
[bt] /usr/local/opt/pvfs-2.6.3/bin/pvfs2-cp(openib_ib_initialize+0x446)
[0x457ca6]
[E 09:24:11.224404] [bt] /usr/local/opt/pvfs-2.6.3/bin/pvfs2-cp
[0x456501]
[E 09:24:11.224433] [bt] /usr/local/opt/pvfs-2.6.3/bin/pvfs2-cp
[0x43cb1e]
[E 09:24:11.224460]
[bt] /usr/local/opt/pvfs-2.6.3/bin/pvfs2-cp(BMI_addr_lookup+0x28a)
[0x43cefa]
[E 09:24:11.224487]
[bt] /usr/local/opt/pvfs-2.6.3/bin/pvfs2-cp(PVFS_isys_fs_add+0x7e)
[0x427f4e]
[E 09:24:11.224513]
[bt] /usr/local/opt/pvfs-2.6.3/bin/pvfs2-cp(PVFS_sys_fs_add+0x17)
[0x4280f7]
[E 09:24:11.224540]
[bt] /usr/local/opt/pvfs-2.6.3/bin/pvfs2-cp(PVFS_util_init_defaults
+0x95) [0x435205]
[E 09:24:11.224570] [bt] /usr/local/opt/pvfs-2.6.3/bin/pvfs2-cp(main
+0x174) [0x40d174]
[E 09:24:11.224596] [bt] /lib64/libc.so.6(__libc_start_main+0xf4)
[0x2b2dd6cde154]
[E 09:24:11.224622] [bt] /usr/local/opt/pvfs-2.6.3/bin/pvfs2-cp
[0x40c699]
It turns out that memorylocked is set to 32K on the shell by default on
SLES 10.
hpcxe005(salmr0)48:limit|grep memorylocked
memorylocked 32 kbytes
Changed this in /etc/security/limits.conf to something bigger and now
everything works fine with pvfs-cp:
hpcxe005(salmr0)37:/usr/local/opt/pvfs-2.6.3/bin/pvfs2-cp
-t /test/pvfs2/testfile1 /test/pvfs2/testfile5
Wrote 655360000 bytes in 2.355496 seconds. 265.336905 MB/seconds
Thanks for the help
Rene
_______________________________________________
Pvfs2-users mailing list
[email protected]
http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users
