Hi Matthieu, To me it looks like the version of pvfs2 commands you're using doesn't know about IB. Can you double check that the version of the pvfs2-ping command used below is linked against OpenIB? You can do that with 'ldd <path to pvfs2-ping>'
On my system, amongst other libraries, I see a reference to libibverbs.so like so: "libibverbs.so.1 => /usr/lib64/libibverbs.so.1 (0x00002b8ba74c0000)" Is that the same for you? If not, post what you get and confirm that 'which pvfs2-ping' points you to the version of pvfs2 that you built against OpenIB. Thanks, Michael On Thu, Mar 24, 2011 at 8:27 AM, Matthieu Dorier < [email protected]> wrote: > Hello, > > Additional informations regarding the problem: doing a pvfs2-ping doesn't > work (neither do all the pvfs2-* commands, such as pvfs2-ls). > I've the following error with pvfs2-ping: > > mdorier@parapluie-31:~$ pvfs2-ping -m /mnt/pvfs2 > > (1) Parsing tab file... > > (2) Initializing system interface... > > (3) Initializing each file system found in tab file: /etc/mtab... > > PVFS2 servers: ib://parapide-5-ib0:3335 > Storage name: pvfs2 > Local mount point: /mnt/pvfs2 > [E 13:01:41.415907] PVFS_isys_fs_add: Failed to initialize any appropriate > BMI methods for addresses: > [E 13:01:41.415970] ib://parapluie-32-ib0:3335 > [E 13:01:41.415992] PVFS_isys_fs_add call: Protocol not available > /mnt/pvfs2: FAILURE! > > Failure: could not initialze at least one of the target file systems. > > (4) Searching for /mnt/pvfs2 in pvfstab... > [E 13:01:41.416022] Error: /mnt/pvfs2/ resides on a PVFS2 file system that > has not yet been initialized. > Failure: could not find filesystem for /mnt/pvfs2 in pvfs2tab /etc/mtab > Entry 0: /mnt/pvfs2 > > The line in /etc/mtab is the following: > > ib://parapide-5-ib0:3335/pvfs2 /mnt/pvfs2 pvfs2 defaults,noauto 0 0 > > What I don't understand is why the kernel module works, but not the pvfs2 > tools or any MPI application? > Thank you for your help, > > Regards > > > Matthieu > > > 2011/3/23 Matthieu Dorier <[email protected]> > >> Actually when I use the standard names (parapide-X or >> parapide-X.rennes.grid5000.fr), the kernel module uses the eth0 >> interface, so I switched again to using the names ending with "-ib0". I >> attach the configuration file. >> >> Matthieu >> >> >> 2011/3/23 Matthieu Dorier <[email protected]> >> >>> Ok actually my nodes are connected to 2 switches: one 1G Eth and one IB, >>> so the names "parapide-X-ib0" were to be used for IP over IB (the real >>> hostnames being "parapide-X", or even "parapide-X.rennes.grid5000.fr", >>> which is the output of the hostname command). I realized it was useless >>> working with the names ending in "-ib0" because I'm not using IP over IB in >>> this case. >>> >>> So I deployed again PVFS using the real hostnames, but I still get the >>> same errors (and again, the kernel module works fine), the configuration >>> line is now: >>> /usr/bin/pvfs2-genconfig --quiet --protocol ib --ioservers "parapide-5" >>> --metaservers "parapide-5" --storage /tmp/pvfs2-storage-space --logfile >>> /tmp/pvfs2-log --fsname pvfs2 pvfs2-fs.conf >>> And the configuration file is attached. >>> >>> I also changed the limits.conf as advised by Kevin Harms, by adding the >>> following line: >>> * hard memlock unlimited >>> But it does change the warning (when I'm logged as user the ulimit -l >>> unlimited says I don't have the permissions). Also tried >>> * hard memlock 1073741824 >>> Doesn't work either. >>> >>> >>> Matthieu >>> >>> >>> 2011/3/22 Becky Ligon <[email protected]> >>> >>>> The kernel module is not required for MPI. >>>> >>>> Becky >>>> -- >>>> Becky Ligon >>>> HPC Admin Staff >>>> PVFS Developer >>>> Clemson University >>>> 864-650-4065 >>>> >>>> > Is make kmod_install also required? >>>> > >>>> > -crispy >>>> > >>>> > >>>> > On 03/22/2011 03:04 PM, Matthieu Dorier wrote: >>>> >> If you mean the command line for pvfs2-genconfig, here it is: >>>> >> >>>> >> /usr/bin/pvfs2-genconfig --quiet --protocol ib --ioservers >>>> >> "parapide-7-ib0" >>>> >> --metaservers "parapide-7-ib0" --storage /tmp/pvfs2-storage-space >>>> >> --logfile >>>> >> /tmp/pvfs2-log --fsname pvfs2 pvfs2-fs.conf >>>> >> >>>> >> I'm basically following the "Quick start guide". >>>> >> >>>> >> If you mean the configuration before compiling, here it is: >>>> >> >>>> >> ./configure --prefix=/usr >>>> >> --with-kernel=/lib/modules/2.6.26-2-amd64/build >>>> >> --with-openib=/usr --without-bmi-tcp >>>> >> (then make, make kmod, make install) >>>> >> >>>> >> Matthieu >>>> >> >>>> >> 2011/3/22 Becky Ligon<[email protected]> >>>> >> >>>> >>> Can you give us the full command line when you configure PVFS? >>>> Seems >>>> >>> that >>>> >>> you are doing this in two steps and must be done in one. >>>> >>> >>>> >>> Becky >>>> >>> -- >>>> >>> Becky Ligon >>>> >>> HPC Admin Staff >>>> >>> PVFS Developer >>>> >>> Clemson University >>>> >>> 864-650-4065 >>>> >>> >>>> >>>> Hello, >>>> >>>> >>>> >>>> I have installed PVFS (version 2.8.2) with InfiniBand support on a >>>> set >>>> >>>> of >>>> >>>> nodes running Debian (linux 2.6). The configuration line being the >>>> >>>> following: >>>> >>>> ./configure --with-openib=/usr --without-bmi-tcp --prefix=/usr >>>> >>>> The filesystem works fine with the kernel module, I'm able to write >>>> >>>> and >>>> >>>> read >>>> >>>> to/from the filesystem. >>>> >>>> Then I compiled Mpich2 (1.2.1p1) with PVFS support using >>>> >>>> --enable-romio >>>> >>>> --with-pvfs2=/usr (among other options which doesn't matter). >>>> >>>> Running the IOR benchmark with POSIX, it works fine, but when using >>>> >>>> MPIIO >>>> >>>> I >>>> >>>> got the following errors (this is from process 0, but all the >>>> >>>> processes >>>> >>>> are >>>> >>>> printing the same) >>>> >>>> >>>> >>>> 0: libibverbs: Warning: RLIMIT_MEMLOCK is 32768 bytes. >>>> >>>> 0: This will severely limit memory registrations. >>>> >>>> 0: [E 14:12:05.314874] Warning: openib_ib_initialize: >>>> ibv_open_device. >>>> >>>> 0: [E 14:12:05.314908] PVFS_isys_fs_add: Failed to initialize any >>>> >>>> appropriate BMI methods for addresses: >>>> >>>> 0: [E 14:12:05.314921] ib:// >>>> parapide-7-ib0.rennes.grid5000.fr:3335 >>>> >>>> 0: [E 14:12:05.314952] PVFS_isys_fs_add call: No such device >>>> >>>> 0: [E 14:12:05.314969] ERROR: could not initialize any file systems >>>> in >>>> >>>> /etc/mtab. >>>> >>>> 0: PVFS_util_init_defaults: No such device (error class: 0) >>>> >>>> >>>> >>>> Do you have any idea where it comes from? I'm unsuccefully trying >>>> to >>>> >>>> reinstall OFED-1.5, recompiling PVFS, Mpich, but I always get the >>>> same >>>> >>>> thing. >>>> >>>> Note: previsouly everything worked well with Ethernet, I was able >>>> to >>>> >>>> have >>>> >>>> kernel module AND MPI-IO working. >>>> >>>> >>>> >>>> Thank you, >>>> >>>> >>>> >>>> Regards >>>> >>>> >>>> >>>> Matthieu Dorier >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> -- >>>> >>>> Matthieu Dorier >>>> >>>> ENS Cachan, Brittany (Computer Science dpt.) >>>> >>>> IRISA Rennes, Office E324 >>>> >>>> http://perso.eleves.bretagne.ens-cachan.fr/~mdori307/wiki/ >>>> >>>> _______________________________________________ >>>> >>>> Pvfs2-users mailing list >>>> >>>> [email protected] >>>> >>>> http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users >>>> >>>> >>>> >>> >>>> >>> >>>> >>> >>>> >> >>>> >> >>>> >> >>>> >> >>>> >> _______________________________________________ >>>> >> Pvfs2-users mailing list >>>> >> [email protected] >>>> >> http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users >>>> > >>>> >>>> >>>> >>> >>> >>> -- >>> Matthieu Dorier >>> ENS Cachan, Brittany (Computer Science dpt.) >>> IRISA Rennes, Office E324 >>> http://perso.eleves.bretagne.ens-cachan.fr/~mdori307/wiki/ >>> >> >> >> >> -- >> Matthieu Dorier >> ENS Cachan, Brittany (Computer Science dpt.) >> IRISA Rennes, Office E324 >> http://perso.eleves.bretagne.ens-cachan.fr/~mdori307/wiki/ >> > > > > -- > Matthieu Dorier > ENS Cachan, Brittany (Computer Science dpt.) > IRISA Rennes, Office E324 > http://perso.eleves.bretagne.ens-cachan.fr/~mdori307/wiki/ > > _______________________________________________ > Pvfs2-users mailing list > [email protected] > http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users > >
_______________________________________________ Pvfs2-users mailing list [email protected] http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users
