Okay, hopefully that's all it is and you're up and running. If not, can you double check that IB is working between the two with something similar to:
login to parapide-5-ib0 and run 'ibv_rc_pingpong' from parapluie-31 run 'ibv_rc_pingpong parapide-5-ib0' Michael On Thu, Mar 24, 2011 at 9:18 AM, Matthieu Dorier < [email protected]> wrote: > Actually I indeed have two version of the pvfs2- tools, I never noticed > that the one called by default was in /usr/local/bin instead of /usr/bin, > I'll check if it works now... > > > 2011/3/24 Matthieu Dorier <[email protected]> > >> Here is what I have: >> >> ldd pvfs2-ping >> linux-vdso.so.1 => (0x00007fff4e106000) >> libpthread.so.0 => /lib/libpthread.so.0 (0x00007fc94f3dd000) >> libibverbs.so.1 => /usr/lib/libibverbs.so.1 (0x00007fc94f1d1000) >> libc.so.6 => /lib/libc.so.6 (0x00007fc94ee7e000) >> /lib64/ld-linux-x86-64.so.2 (0x00007fc94f5f9000) >> libdl.so.2 => /lib/libdl.so.2 (0x00007fc94ec7a000) >> >> Matthieu >> >> >> 2011/3/24 Michael Moore <[email protected]> >> >>> Hi Matthieu, >>> >>> To me it looks like the version of pvfs2 commands you're using doesn't >>> know about IB. Can you double check that the version of the pvfs2-ping >>> command used below is linked against OpenIB? You can do that with 'ldd <path >>> to pvfs2-ping>' >>> >>> On my system, amongst other libraries, I see a reference to libibverbs.so >>> like so: >>> "libibverbs.so.1 => /usr/lib64/libibverbs.so.1 (0x00002b8ba74c0000)" >>> >>> Is that the same for you? If not, post what you get and confirm that >>> 'which pvfs2-ping' points you to the version of pvfs2 that you built against >>> OpenIB. >>> >>> Thanks, >>> Michael >>> >>> >>> On Thu, Mar 24, 2011 at 8:27 AM, Matthieu Dorier < >>> [email protected]> wrote: >>> >>>> Hello, >>>> >>>> Additional informations regarding the problem: doing a pvfs2-ping >>>> doesn't work (neither do all the pvfs2-* commands, such as pvfs2-ls). >>>> I've the following error with pvfs2-ping: >>>> >>>> mdorier@parapluie-31:~$ pvfs2-ping -m /mnt/pvfs2 >>>> >>>> (1) Parsing tab file... >>>> >>>> (2) Initializing system interface... >>>> >>>> (3) Initializing each file system found in tab file: /etc/mtab... >>>> >>>> PVFS2 servers: ib://parapide-5-ib0:3335 >>>> Storage name: pvfs2 >>>> Local mount point: /mnt/pvfs2 >>>> [E 13:01:41.415907] PVFS_isys_fs_add: Failed to initialize any >>>> appropriate BMI methods for addresses: >>>> [E 13:01:41.415970] ib://parapluie-32-ib0:3335 >>>> [E 13:01:41.415992] PVFS_isys_fs_add call: Protocol not available >>>> /mnt/pvfs2: FAILURE! >>>> >>>> Failure: could not initialze at least one of the target file systems. >>>> >>>> (4) Searching for /mnt/pvfs2 in pvfstab... >>>> [E 13:01:41.416022] Error: /mnt/pvfs2/ resides on a PVFS2 file system >>>> that has not yet been initialized. >>>> Failure: could not find filesystem for /mnt/pvfs2 in pvfs2tab /etc/mtab >>>> Entry 0: /mnt/pvfs2 >>>> >>>> The line in /etc/mtab is the following: >>>> >>>> ib://parapide-5-ib0:3335/pvfs2 /mnt/pvfs2 pvfs2 defaults,noauto 0 0 >>>> >>>> What I don't understand is why the kernel module works, but not the >>>> pvfs2 tools or any MPI application? >>>> Thank you for your help, >>>> >>>> Regards >>>> >>>> >>>> Matthieu >>>> >>>> >>>> 2011/3/23 Matthieu Dorier < >>>> [email protected]> >>>> >>>>> Actually when I use the standard names (parapide-X or >>>>> parapide-X.rennes.grid5000.fr), the kernel module uses the eth0 >>>>> interface, so I switched again to using the names ending with "-ib0". I >>>>> attach the configuration file. >>>>> >>>>> Matthieu >>>>> >>>>> >>>>> 2011/3/23 Matthieu Dorier < >>>>> [email protected]> >>>>> >>>>>> Ok actually my nodes are connected to 2 switches: one 1G Eth and one >>>>>> IB, so the names "parapide-X-ib0" were to be used for IP over IB (the >>>>>> real >>>>>> hostnames being "parapide-X", or even "parapide-X.rennes.grid5000.fr", >>>>>> which is the output of the hostname command). I realized it was useless >>>>>> working with the names ending in "-ib0" because I'm not using IP over IB >>>>>> in >>>>>> this case. >>>>>> >>>>>> So I deployed again PVFS using the real hostnames, but I still get the >>>>>> same errors (and again, the kernel module works fine), the configuration >>>>>> line is now: >>>>>> /usr/bin/pvfs2-genconfig --quiet --protocol ib --ioservers >>>>>> "parapide-5" --metaservers "parapide-5" --storage >>>>>> /tmp/pvfs2-storage-space >>>>>> --logfile /tmp/pvfs2-log --fsname pvfs2 pvfs2-fs.conf >>>>>> And the configuration file is attached. >>>>>> >>>>>> I also changed the limits.conf as advised by Kevin Harms, by adding >>>>>> the following line: >>>>>> * hard memlock unlimited >>>>>> But it does change the warning (when I'm logged as user the ulimit -l >>>>>> unlimited says I don't have the permissions). Also tried >>>>>> * hard memlock 1073741824 >>>>>> Doesn't work either. >>>>>> >>>>>> >>>>>> Matthieu >>>>>> >>>>>> >>>>>> 2011/3/22 Becky Ligon <[email protected]> >>>>>> >>>>>>> The kernel module is not required for MPI. >>>>>>> >>>>>>> Becky >>>>>>> -- >>>>>>> Becky Ligon >>>>>>> HPC Admin Staff >>>>>>> PVFS Developer >>>>>>> Clemson University >>>>>>> 864-650-4065 >>>>>>> >>>>>>> > Is make kmod_install also required? >>>>>>> > >>>>>>> > -crispy >>>>>>> > >>>>>>> > >>>>>>> > On 03/22/2011 03:04 PM, Matthieu Dorier wrote: >>>>>>> >> If you mean the command line for pvfs2-genconfig, here it is: >>>>>>> >> >>>>>>> >> /usr/bin/pvfs2-genconfig --quiet --protocol ib --ioservers >>>>>>> >> "parapide-7-ib0" >>>>>>> >> --metaservers "parapide-7-ib0" --storage /tmp/pvfs2-storage-space >>>>>>> >> --logfile >>>>>>> >> /tmp/pvfs2-log --fsname pvfs2 pvfs2-fs.conf >>>>>>> >> >>>>>>> >> I'm basically following the "Quick start guide". >>>>>>> >> >>>>>>> >> If you mean the configuration before compiling, here it is: >>>>>>> >> >>>>>>> >> ./configure --prefix=/usr >>>>>>> >> --with-kernel=/lib/modules/2.6.26-2-amd64/build >>>>>>> >> --with-openib=/usr --without-bmi-tcp >>>>>>> >> (then make, make kmod, make install) >>>>>>> >> >>>>>>> >> Matthieu >>>>>>> >> >>>>>>> >> 2011/3/22 Becky Ligon<[email protected]> >>>>>>> >> >>>>>>> >>> Can you give us the full command line when you configure PVFS? >>>>>>> Seems >>>>>>> >>> that >>>>>>> >>> you are doing this in two steps and must be done in one. >>>>>>> >>> >>>>>>> >>> Becky >>>>>>> >>> -- >>>>>>> >>> Becky Ligon >>>>>>> >>> HPC Admin Staff >>>>>>> >>> PVFS Developer >>>>>>> >>> Clemson University >>>>>>> >>> 864-650-4065 >>>>>>> >>> >>>>>>> >>>> Hello, >>>>>>> >>>> >>>>>>> >>>> I have installed PVFS (version 2.8.2) with InfiniBand support on >>>>>>> a set >>>>>>> >>>> of >>>>>>> >>>> nodes running Debian (linux 2.6). The configuration line being >>>>>>> the >>>>>>> >>>> following: >>>>>>> >>>> ./configure --with-openib=/usr --without-bmi-tcp --prefix=/usr >>>>>>> >>>> The filesystem works fine with the kernel module, I'm able to >>>>>>> write >>>>>>> >>>> and >>>>>>> >>>> read >>>>>>> >>>> to/from the filesystem. >>>>>>> >>>> Then I compiled Mpich2 (1.2.1p1) with PVFS support using >>>>>>> >>>> --enable-romio >>>>>>> >>>> --with-pvfs2=/usr (among other options which doesn't matter). >>>>>>> >>>> Running the IOR benchmark with POSIX, it works fine, but when >>>>>>> using >>>>>>> >>>> MPIIO >>>>>>> >>>> I >>>>>>> >>>> got the following errors (this is from process 0, but all the >>>>>>> >>>> processes >>>>>>> >>>> are >>>>>>> >>>> printing the same) >>>>>>> >>>> >>>>>>> >>>> 0: libibverbs: Warning: RLIMIT_MEMLOCK is 32768 bytes. >>>>>>> >>>> 0: This will severely limit memory registrations. >>>>>>> >>>> 0: [E 14:12:05.314874] Warning: openib_ib_initialize: >>>>>>> ibv_open_device. >>>>>>> >>>> 0: [E 14:12:05.314908] PVFS_isys_fs_add: Failed to initialize >>>>>>> any >>>>>>> >>>> appropriate BMI methods for addresses: >>>>>>> >>>> 0: [E 14:12:05.314921] ib:// >>>>>>> parapide-7-ib0.rennes.grid5000.fr:3335 >>>>>>> >>>> 0: [E 14:12:05.314952] PVFS_isys_fs_add call: No such device >>>>>>> >>>> 0: [E 14:12:05.314969] ERROR: could not initialize any file >>>>>>> systems in >>>>>>> >>>> /etc/mtab. >>>>>>> >>>> 0: PVFS_util_init_defaults: No such device (error class: 0) >>>>>>> >>>> >>>>>>> >>>> Do you have any idea where it comes from? I'm unsuccefully >>>>>>> trying to >>>>>>> >>>> reinstall OFED-1.5, recompiling PVFS, Mpich, but I always get >>>>>>> the same >>>>>>> >>>> thing. >>>>>>> >>>> Note: previsouly everything worked well with Ethernet, I was >>>>>>> able to >>>>>>> >>>> have >>>>>>> >>>> kernel module AND MPI-IO working. >>>>>>> >>>> >>>>>>> >>>> Thank you, >>>>>>> >>>> >>>>>>> >>>> Regards >>>>>>> >>>> >>>>>>> >>>> Matthieu Dorier >>>>>>> >>>> >>>>>>> >>>> >>>>>>> >>>> >>>>>>> >>>> -- >>>>>>> >>>> Matthieu Dorier >>>>>>> >>>> ENS Cachan, Brittany (Computer Science dpt.) >>>>>>> >>>> IRISA Rennes, Office E324 >>>>>>> >>>> http://perso.eleves.bretagne.ens-cachan.fr/~mdori307/wiki/ >>>>>>> >>>> _______________________________________________ >>>>>>> >>>> Pvfs2-users mailing list >>>>>>> >>>> [email protected] >>>>>>> >>>> http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users >>>>>>> >>>> >>>>>>> >>> >>>>>>> >>> >>>>>>> >>> >>>>>>> >> >>>>>>> >> >>>>>>> >> >>>>>>> >> >>>>>>> >> _______________________________________________ >>>>>>> >> Pvfs2-users mailing list >>>>>>> >> [email protected] >>>>>>> >> http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users >>>>>>> > >>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Matthieu Dorier >>>>>> ENS Cachan, Brittany (Computer Science dpt.) >>>>>> IRISA Rennes, Office E324 >>>>>> http://perso.eleves.bretagne.ens-cachan.fr/~mdori307/wiki/ >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> Matthieu Dorier >>>>> ENS Cachan, Brittany (Computer Science dpt.) >>>>> IRISA Rennes, Office E324 >>>>> http://perso.eleves.bretagne.ens-cachan.fr/~mdori307/wiki/ >>>>> >>>> >>>> >>>> >>>> -- >>>> Matthieu Dorier >>>> ENS Cachan, Brittany (Computer Science dpt.) >>>> IRISA Rennes, Office E324 >>>> http://perso.eleves.bretagne.ens-cachan.fr/~mdori307/wiki/ >>>> >>>> _______________________________________________ >>>> Pvfs2-users mailing list >>>> [email protected] >>>> http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users >>>> >>>> >>> >> >> >> -- >> Matthieu Dorier >> ENS Cachan, Brittany (Computer Science dpt.) >> IRISA Rennes, Office E324 >> http://perso.eleves.bretagne.ens-cachan.fr/~mdori307/wiki/ >> > > > > -- > Matthieu Dorier > ENS Cachan, Brittany (Computer Science dpt.) > IRISA Rennes, Office E324 > http://perso.eleves.bretagne.ens-cachan.fr/~mdori307/wiki/ >
_______________________________________________ Pvfs2-users mailing list [email protected] http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users
