Hello, Additional informations regarding the problem: doing a pvfs2-ping doesn't work (neither do all the pvfs2-* commands, such as pvfs2-ls). I've the following error with pvfs2-ping:
mdorier@parapluie-31:~$ pvfs2-ping -m /mnt/pvfs2 (1) Parsing tab file... (2) Initializing system interface... (3) Initializing each file system found in tab file: /etc/mtab... PVFS2 servers: ib://parapide-5-ib0:3335 Storage name: pvfs2 Local mount point: /mnt/pvfs2 [E 13:01:41.415907] PVFS_isys_fs_add: Failed to initialize any appropriate BMI methods for addresses: [E 13:01:41.415970] ib://parapluie-32-ib0:3335 [E 13:01:41.415992] PVFS_isys_fs_add call: Protocol not available /mnt/pvfs2: FAILURE! Failure: could not initialze at least one of the target file systems. (4) Searching for /mnt/pvfs2 in pvfstab... [E 13:01:41.416022] Error: /mnt/pvfs2/ resides on a PVFS2 file system that has not yet been initialized. Failure: could not find filesystem for /mnt/pvfs2 in pvfs2tab /etc/mtab Entry 0: /mnt/pvfs2 The line in /etc/mtab is the following: ib://parapide-5-ib0:3335/pvfs2 /mnt/pvfs2 pvfs2 defaults,noauto 0 0 What I don't understand is why the kernel module works, but not the pvfs2 tools or any MPI application? Thank you for your help, Regards Matthieu 2011/3/23 Matthieu Dorier <[email protected]> > Actually when I use the standard names (parapide-X or > parapide-X.rennes.grid5000.fr), the kernel module uses the eth0 interface, > so I switched again to using the names ending with "-ib0". I attach the > configuration file. > > Matthieu > > > 2011/3/23 Matthieu Dorier <[email protected]> > >> Ok actually my nodes are connected to 2 switches: one 1G Eth and one IB, >> so the names "parapide-X-ib0" were to be used for IP over IB (the real >> hostnames being "parapide-X", or even "parapide-X.rennes.grid5000.fr", >> which is the output of the hostname command). I realized it was useless >> working with the names ending in "-ib0" because I'm not using IP over IB in >> this case. >> >> So I deployed again PVFS using the real hostnames, but I still get the >> same errors (and again, the kernel module works fine), the configuration >> line is now: >> /usr/bin/pvfs2-genconfig --quiet --protocol ib --ioservers "parapide-5" >> --metaservers "parapide-5" --storage /tmp/pvfs2-storage-space --logfile >> /tmp/pvfs2-log --fsname pvfs2 pvfs2-fs.conf >> And the configuration file is attached. >> >> I also changed the limits.conf as advised by Kevin Harms, by adding the >> following line: >> * hard memlock unlimited >> But it does change the warning (when I'm logged as user the ulimit -l >> unlimited says I don't have the permissions). Also tried >> * hard memlock 1073741824 >> Doesn't work either. >> >> >> Matthieu >> >> >> 2011/3/22 Becky Ligon <[email protected]> >> >>> The kernel module is not required for MPI. >>> >>> Becky >>> -- >>> Becky Ligon >>> HPC Admin Staff >>> PVFS Developer >>> Clemson University >>> 864-650-4065 >>> >>> > Is make kmod_install also required? >>> > >>> > -crispy >>> > >>> > >>> > On 03/22/2011 03:04 PM, Matthieu Dorier wrote: >>> >> If you mean the command line for pvfs2-genconfig, here it is: >>> >> >>> >> /usr/bin/pvfs2-genconfig --quiet --protocol ib --ioservers >>> >> "parapide-7-ib0" >>> >> --metaservers "parapide-7-ib0" --storage /tmp/pvfs2-storage-space >>> >> --logfile >>> >> /tmp/pvfs2-log --fsname pvfs2 pvfs2-fs.conf >>> >> >>> >> I'm basically following the "Quick start guide". >>> >> >>> >> If you mean the configuration before compiling, here it is: >>> >> >>> >> ./configure --prefix=/usr >>> >> --with-kernel=/lib/modules/2.6.26-2-amd64/build >>> >> --with-openib=/usr --without-bmi-tcp >>> >> (then make, make kmod, make install) >>> >> >>> >> Matthieu >>> >> >>> >> 2011/3/22 Becky Ligon<[email protected]> >>> >> >>> >>> Can you give us the full command line when you configure PVFS? Seems >>> >>> that >>> >>> you are doing this in two steps and must be done in one. >>> >>> >>> >>> Becky >>> >>> -- >>> >>> Becky Ligon >>> >>> HPC Admin Staff >>> >>> PVFS Developer >>> >>> Clemson University >>> >>> 864-650-4065 >>> >>> >>> >>>> Hello, >>> >>>> >>> >>>> I have installed PVFS (version 2.8.2) with InfiniBand support on a >>> set >>> >>>> of >>> >>>> nodes running Debian (linux 2.6). The configuration line being the >>> >>>> following: >>> >>>> ./configure --with-openib=/usr --without-bmi-tcp --prefix=/usr >>> >>>> The filesystem works fine with the kernel module, I'm able to write >>> >>>> and >>> >>>> read >>> >>>> to/from the filesystem. >>> >>>> Then I compiled Mpich2 (1.2.1p1) with PVFS support using >>> >>>> --enable-romio >>> >>>> --with-pvfs2=/usr (among other options which doesn't matter). >>> >>>> Running the IOR benchmark with POSIX, it works fine, but when using >>> >>>> MPIIO >>> >>>> I >>> >>>> got the following errors (this is from process 0, but all the >>> >>>> processes >>> >>>> are >>> >>>> printing the same) >>> >>>> >>> >>>> 0: libibverbs: Warning: RLIMIT_MEMLOCK is 32768 bytes. >>> >>>> 0: This will severely limit memory registrations. >>> >>>> 0: [E 14:12:05.314874] Warning: openib_ib_initialize: >>> ibv_open_device. >>> >>>> 0: [E 14:12:05.314908] PVFS_isys_fs_add: Failed to initialize any >>> >>>> appropriate BMI methods for addresses: >>> >>>> 0: [E 14:12:05.314921] ib:// >>> parapide-7-ib0.rennes.grid5000.fr:3335 >>> >>>> 0: [E 14:12:05.314952] PVFS_isys_fs_add call: No such device >>> >>>> 0: [E 14:12:05.314969] ERROR: could not initialize any file systems >>> in >>> >>>> /etc/mtab. >>> >>>> 0: PVFS_util_init_defaults: No such device (error class: 0) >>> >>>> >>> >>>> Do you have any idea where it comes from? I'm unsuccefully trying to >>> >>>> reinstall OFED-1.5, recompiling PVFS, Mpich, but I always get the >>> same >>> >>>> thing. >>> >>>> Note: previsouly everything worked well with Ethernet, I was able to >>> >>>> have >>> >>>> kernel module AND MPI-IO working. >>> >>>> >>> >>>> Thank you, >>> >>>> >>> >>>> Regards >>> >>>> >>> >>>> Matthieu Dorier >>> >>>> >>> >>>> >>> >>>> >>> >>>> -- >>> >>>> Matthieu Dorier >>> >>>> ENS Cachan, Brittany (Computer Science dpt.) >>> >>>> IRISA Rennes, Office E324 >>> >>>> http://perso.eleves.bretagne.ens-cachan.fr/~mdori307/wiki/ >>> >>>> _______________________________________________ >>> >>>> Pvfs2-users mailing list >>> >>>> [email protected] >>> >>>> http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users >>> >>>> >>> >>> >>> >>> >>> >>> >>> >> >>> >> >>> >> >>> >> >>> >> _______________________________________________ >>> >> Pvfs2-users mailing list >>> >> [email protected] >>> >> http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users >>> > >>> >>> >>> >> >> >> -- >> Matthieu Dorier >> ENS Cachan, Brittany (Computer Science dpt.) >> IRISA Rennes, Office E324 >> http://perso.eleves.bretagne.ens-cachan.fr/~mdori307/wiki/ >> > > > > -- > Matthieu Dorier > ENS Cachan, Brittany (Computer Science dpt.) > IRISA Rennes, Office E324 > http://perso.eleves.bretagne.ens-cachan.fr/~mdori307/wiki/ > -- Matthieu Dorier ENS Cachan, Brittany (Computer Science dpt.) IRISA Rennes, Office E324 http://perso.eleves.bretagne.ens-cachan.fr/~mdori307/wiki/
_______________________________________________ Pvfs2-users mailing list [email protected] http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users
