Ok actually my nodes are connected to 2 switches: one 1G Eth and one IB, so the names "parapide-X-ib0" were to be used for IP over IB (the real hostnames being "parapide-X", or even "parapide-X.rennes.grid5000.fr", which is the output of the hostname command). I realized it was useless working with the names ending in "-ib0" because I'm not using IP over IB in this case.
So I deployed again PVFS using the real hostnames, but I still get the same errors (and again, the kernel module works fine), the configuration line is now: /usr/bin/pvfs2-genconfig --quiet --protocol ib --ioservers "parapide-5" --metaservers "parapide-5" --storage /tmp/pvfs2-storage-space --logfile /tmp/pvfs2-log --fsname pvfs2 pvfs2-fs.conf And the configuration file is attached. I also changed the limits.conf as advised by Kevin Harms, by adding the following line: * hard memlock unlimited But it does change the warning (when I'm logged as user the ulimit -l unlimited says I don't have the permissions). Also tried * hard memlock 1073741824 Doesn't work either. Matthieu 2011/3/22 Becky Ligon <[email protected]> > The kernel module is not required for MPI. > > Becky > -- > Becky Ligon > HPC Admin Staff > PVFS Developer > Clemson University > 864-650-4065 > > > Is make kmod_install also required? > > > > -crispy > > > > > > On 03/22/2011 03:04 PM, Matthieu Dorier wrote: > >> If you mean the command line for pvfs2-genconfig, here it is: > >> > >> /usr/bin/pvfs2-genconfig --quiet --protocol ib --ioservers > >> "parapide-7-ib0" > >> --metaservers "parapide-7-ib0" --storage /tmp/pvfs2-storage-space > >> --logfile > >> /tmp/pvfs2-log --fsname pvfs2 pvfs2-fs.conf > >> > >> I'm basically following the "Quick start guide". > >> > >> If you mean the configuration before compiling, here it is: > >> > >> ./configure --prefix=/usr > >> --with-kernel=/lib/modules/2.6.26-2-amd64/build > >> --with-openib=/usr --without-bmi-tcp > >> (then make, make kmod, make install) > >> > >> Matthieu > >> > >> 2011/3/22 Becky Ligon<[email protected]> > >> > >>> Can you give us the full command line when you configure PVFS? Seems > >>> that > >>> you are doing this in two steps and must be done in one. > >>> > >>> Becky > >>> -- > >>> Becky Ligon > >>> HPC Admin Staff > >>> PVFS Developer > >>> Clemson University > >>> 864-650-4065 > >>> > >>>> Hello, > >>>> > >>>> I have installed PVFS (version 2.8.2) with InfiniBand support on a set > >>>> of > >>>> nodes running Debian (linux 2.6). The configuration line being the > >>>> following: > >>>> ./configure --with-openib=/usr --without-bmi-tcp --prefix=/usr > >>>> The filesystem works fine with the kernel module, I'm able to write > >>>> and > >>>> read > >>>> to/from the filesystem. > >>>> Then I compiled Mpich2 (1.2.1p1) with PVFS support using > >>>> --enable-romio > >>>> --with-pvfs2=/usr (among other options which doesn't matter). > >>>> Running the IOR benchmark with POSIX, it works fine, but when using > >>>> MPIIO > >>>> I > >>>> got the following errors (this is from process 0, but all the > >>>> processes > >>>> are > >>>> printing the same) > >>>> > >>>> 0: libibverbs: Warning: RLIMIT_MEMLOCK is 32768 bytes. > >>>> 0: This will severely limit memory registrations. > >>>> 0: [E 14:12:05.314874] Warning: openib_ib_initialize: ibv_open_device. > >>>> 0: [E 14:12:05.314908] PVFS_isys_fs_add: Failed to initialize any > >>>> appropriate BMI methods for addresses: > >>>> 0: [E 14:12:05.314921] ib:// > parapide-7-ib0.rennes.grid5000.fr:3335 > >>>> 0: [E 14:12:05.314952] PVFS_isys_fs_add call: No such device > >>>> 0: [E 14:12:05.314969] ERROR: could not initialize any file systems in > >>>> /etc/mtab. > >>>> 0: PVFS_util_init_defaults: No such device (error class: 0) > >>>> > >>>> Do you have any idea where it comes from? I'm unsuccefully trying to > >>>> reinstall OFED-1.5, recompiling PVFS, Mpich, but I always get the same > >>>> thing. > >>>> Note: previsouly everything worked well with Ethernet, I was able to > >>>> have > >>>> kernel module AND MPI-IO working. > >>>> > >>>> Thank you, > >>>> > >>>> Regards > >>>> > >>>> Matthieu Dorier > >>>> > >>>> > >>>> > >>>> -- > >>>> Matthieu Dorier > >>>> ENS Cachan, Brittany (Computer Science dpt.) > >>>> IRISA Rennes, Office E324 > >>>> http://perso.eleves.bretagne.ens-cachan.fr/~mdori307/wiki/ > >>>> _______________________________________________ > >>>> Pvfs2-users mailing list > >>>> [email protected] > >>>> http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users > >>>> > >>> > >>> > >>> > >> > >> > >> > >> > >> _______________________________________________ > >> Pvfs2-users mailing list > >> [email protected] > >> http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users > > > > > -- Matthieu Dorier ENS Cachan, Brittany (Computer Science dpt.) IRISA Rennes, Office E324 http://perso.eleves.bretagne.ens-cachan.fr/~mdori307/wiki/
pvfs2-fs.conf
Description: Binary data
_______________________________________________ Pvfs2-users mailing list [email protected] http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users
