Actually I indeed have two version of the pvfs2- tools, I never noticed that the one called by default was in /usr/local/bin instead of /usr/bin, I'll check if it works now...
2011/3/24 Matthieu Dorier <[email protected]> > Here is what I have: > > ldd pvfs2-ping > linux-vdso.so.1 => (0x00007fff4e106000) > libpthread.so.0 => /lib/libpthread.so.0 (0x00007fc94f3dd000) > libibverbs.so.1 => /usr/lib/libibverbs.so.1 (0x00007fc94f1d1000) > libc.so.6 => /lib/libc.so.6 (0x00007fc94ee7e000) > /lib64/ld-linux-x86-64.so.2 (0x00007fc94f5f9000) > libdl.so.2 => /lib/libdl.so.2 (0x00007fc94ec7a000) > > Matthieu > > > 2011/3/24 Michael Moore <[email protected]> > >> Hi Matthieu, >> >> To me it looks like the version of pvfs2 commands you're using doesn't >> know about IB. Can you double check that the version of the pvfs2-ping >> command used below is linked against OpenIB? You can do that with 'ldd <path >> to pvfs2-ping>' >> >> On my system, amongst other libraries, I see a reference to libibverbs.so >> like so: >> "libibverbs.so.1 => /usr/lib64/libibverbs.so.1 (0x00002b8ba74c0000)" >> >> Is that the same for you? If not, post what you get and confirm that >> 'which pvfs2-ping' points you to the version of pvfs2 that you built against >> OpenIB. >> >> Thanks, >> Michael >> >> >> On Thu, Mar 24, 2011 at 8:27 AM, Matthieu Dorier < >> [email protected]> wrote: >> >>> Hello, >>> >>> Additional informations regarding the problem: doing a pvfs2-ping doesn't >>> work (neither do all the pvfs2-* commands, such as pvfs2-ls). >>> I've the following error with pvfs2-ping: >>> >>> mdorier@parapluie-31:~$ pvfs2-ping -m /mnt/pvfs2 >>> >>> (1) Parsing tab file... >>> >>> (2) Initializing system interface... >>> >>> (3) Initializing each file system found in tab file: /etc/mtab... >>> >>> PVFS2 servers: ib://parapide-5-ib0:3335 >>> Storage name: pvfs2 >>> Local mount point: /mnt/pvfs2 >>> [E 13:01:41.415907] PVFS_isys_fs_add: Failed to initialize any >>> appropriate BMI methods for addresses: >>> [E 13:01:41.415970] ib://parapluie-32-ib0:3335 >>> [E 13:01:41.415992] PVFS_isys_fs_add call: Protocol not available >>> /mnt/pvfs2: FAILURE! >>> >>> Failure: could not initialze at least one of the target file systems. >>> >>> (4) Searching for /mnt/pvfs2 in pvfstab... >>> [E 13:01:41.416022] Error: /mnt/pvfs2/ resides on a PVFS2 file system >>> that has not yet been initialized. >>> Failure: could not find filesystem for /mnt/pvfs2 in pvfs2tab /etc/mtab >>> Entry 0: /mnt/pvfs2 >>> >>> The line in /etc/mtab is the following: >>> >>> ib://parapide-5-ib0:3335/pvfs2 /mnt/pvfs2 pvfs2 defaults,noauto 0 0 >>> >>> What I don't understand is why the kernel module works, but not the pvfs2 >>> tools or any MPI application? >>> Thank you for your help, >>> >>> Regards >>> >>> >>> Matthieu >>> >>> >>> 2011/3/23 Matthieu Dorier <[email protected] >>> > >>> >>>> Actually when I use the standard names (parapide-X or >>>> parapide-X.rennes.grid5000.fr), the kernel module uses the eth0 >>>> interface, so I switched again to using the names ending with "-ib0". I >>>> attach the configuration file. >>>> >>>> Matthieu >>>> >>>> >>>> 2011/3/23 Matthieu Dorier < >>>> [email protected]> >>>> >>>>> Ok actually my nodes are connected to 2 switches: one 1G Eth and one >>>>> IB, so the names "parapide-X-ib0" were to be used for IP over IB (the real >>>>> hostnames being "parapide-X", or even "parapide-X.rennes.grid5000.fr", >>>>> which is the output of the hostname command). I realized it was useless >>>>> working with the names ending in "-ib0" because I'm not using IP over IB >>>>> in >>>>> this case. >>>>> >>>>> So I deployed again PVFS using the real hostnames, but I still get the >>>>> same errors (and again, the kernel module works fine), the configuration >>>>> line is now: >>>>> /usr/bin/pvfs2-genconfig --quiet --protocol ib --ioservers "parapide-5" >>>>> --metaservers "parapide-5" --storage /tmp/pvfs2-storage-space --logfile >>>>> /tmp/pvfs2-log --fsname pvfs2 pvfs2-fs.conf >>>>> And the configuration file is attached. >>>>> >>>>> I also changed the limits.conf as advised by Kevin Harms, by adding the >>>>> following line: >>>>> * hard memlock unlimited >>>>> But it does change the warning (when I'm logged as user the ulimit -l >>>>> unlimited says I don't have the permissions). Also tried >>>>> * hard memlock 1073741824 >>>>> Doesn't work either. >>>>> >>>>> >>>>> Matthieu >>>>> >>>>> >>>>> 2011/3/22 Becky Ligon <[email protected]> >>>>> >>>>>> The kernel module is not required for MPI. >>>>>> >>>>>> Becky >>>>>> -- >>>>>> Becky Ligon >>>>>> HPC Admin Staff >>>>>> PVFS Developer >>>>>> Clemson University >>>>>> 864-650-4065 >>>>>> >>>>>> > Is make kmod_install also required? >>>>>> > >>>>>> > -crispy >>>>>> > >>>>>> > >>>>>> > On 03/22/2011 03:04 PM, Matthieu Dorier wrote: >>>>>> >> If you mean the command line for pvfs2-genconfig, here it is: >>>>>> >> >>>>>> >> /usr/bin/pvfs2-genconfig --quiet --protocol ib --ioservers >>>>>> >> "parapide-7-ib0" >>>>>> >> --metaservers "parapide-7-ib0" --storage /tmp/pvfs2-storage-space >>>>>> >> --logfile >>>>>> >> /tmp/pvfs2-log --fsname pvfs2 pvfs2-fs.conf >>>>>> >> >>>>>> >> I'm basically following the "Quick start guide". >>>>>> >> >>>>>> >> If you mean the configuration before compiling, here it is: >>>>>> >> >>>>>> >> ./configure --prefix=/usr >>>>>> >> --with-kernel=/lib/modules/2.6.26-2-amd64/build >>>>>> >> --with-openib=/usr --without-bmi-tcp >>>>>> >> (then make, make kmod, make install) >>>>>> >> >>>>>> >> Matthieu >>>>>> >> >>>>>> >> 2011/3/22 Becky Ligon<[email protected]> >>>>>> >> >>>>>> >>> Can you give us the full command line when you configure PVFS? >>>>>> Seems >>>>>> >>> that >>>>>> >>> you are doing this in two steps and must be done in one. >>>>>> >>> >>>>>> >>> Becky >>>>>> >>> -- >>>>>> >>> Becky Ligon >>>>>> >>> HPC Admin Staff >>>>>> >>> PVFS Developer >>>>>> >>> Clemson University >>>>>> >>> 864-650-4065 >>>>>> >>> >>>>>> >>>> Hello, >>>>>> >>>> >>>>>> >>>> I have installed PVFS (version 2.8.2) with InfiniBand support on >>>>>> a set >>>>>> >>>> of >>>>>> >>>> nodes running Debian (linux 2.6). The configuration line being >>>>>> the >>>>>> >>>> following: >>>>>> >>>> ./configure --with-openib=/usr --without-bmi-tcp --prefix=/usr >>>>>> >>>> The filesystem works fine with the kernel module, I'm able to >>>>>> write >>>>>> >>>> and >>>>>> >>>> read >>>>>> >>>> to/from the filesystem. >>>>>> >>>> Then I compiled Mpich2 (1.2.1p1) with PVFS support using >>>>>> >>>> --enable-romio >>>>>> >>>> --with-pvfs2=/usr (among other options which doesn't matter). >>>>>> >>>> Running the IOR benchmark with POSIX, it works fine, but when >>>>>> using >>>>>> >>>> MPIIO >>>>>> >>>> I >>>>>> >>>> got the following errors (this is from process 0, but all the >>>>>> >>>> processes >>>>>> >>>> are >>>>>> >>>> printing the same) >>>>>> >>>> >>>>>> >>>> 0: libibverbs: Warning: RLIMIT_MEMLOCK is 32768 bytes. >>>>>> >>>> 0: This will severely limit memory registrations. >>>>>> >>>> 0: [E 14:12:05.314874] Warning: openib_ib_initialize: >>>>>> ibv_open_device. >>>>>> >>>> 0: [E 14:12:05.314908] PVFS_isys_fs_add: Failed to initialize any >>>>>> >>>> appropriate BMI methods for addresses: >>>>>> >>>> 0: [E 14:12:05.314921] ib:// >>>>>> parapide-7-ib0.rennes.grid5000.fr:3335 >>>>>> >>>> 0: [E 14:12:05.314952] PVFS_isys_fs_add call: No such device >>>>>> >>>> 0: [E 14:12:05.314969] ERROR: could not initialize any file >>>>>> systems in >>>>>> >>>> /etc/mtab. >>>>>> >>>> 0: PVFS_util_init_defaults: No such device (error class: 0) >>>>>> >>>> >>>>>> >>>> Do you have any idea where it comes from? I'm unsuccefully trying >>>>>> to >>>>>> >>>> reinstall OFED-1.5, recompiling PVFS, Mpich, but I always get the >>>>>> same >>>>>> >>>> thing. >>>>>> >>>> Note: previsouly everything worked well with Ethernet, I was able >>>>>> to >>>>>> >>>> have >>>>>> >>>> kernel module AND MPI-IO working. >>>>>> >>>> >>>>>> >>>> Thank you, >>>>>> >>>> >>>>>> >>>> Regards >>>>>> >>>> >>>>>> >>>> Matthieu Dorier >>>>>> >>>> >>>>>> >>>> >>>>>> >>>> >>>>>> >>>> -- >>>>>> >>>> Matthieu Dorier >>>>>> >>>> ENS Cachan, Brittany (Computer Science dpt.) >>>>>> >>>> IRISA Rennes, Office E324 >>>>>> >>>> http://perso.eleves.bretagne.ens-cachan.fr/~mdori307/wiki/ >>>>>> >>>> _______________________________________________ >>>>>> >>>> Pvfs2-users mailing list >>>>>> >>>> [email protected] >>>>>> >>>> http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users >>>>>> >>>> >>>>>> >>> >>>>>> >>> >>>>>> >>> >>>>>> >> >>>>>> >> >>>>>> >> >>>>>> >> >>>>>> >> _______________________________________________ >>>>>> >> Pvfs2-users mailing list >>>>>> >> [email protected] >>>>>> >> http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users >>>>>> > >>>>>> >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> Matthieu Dorier >>>>> ENS Cachan, Brittany (Computer Science dpt.) >>>>> IRISA Rennes, Office E324 >>>>> http://perso.eleves.bretagne.ens-cachan.fr/~mdori307/wiki/ >>>>> >>>> >>>> >>>> >>>> -- >>>> Matthieu Dorier >>>> ENS Cachan, Brittany (Computer Science dpt.) >>>> IRISA Rennes, Office E324 >>>> http://perso.eleves.bretagne.ens-cachan.fr/~mdori307/wiki/ >>>> >>> >>> >>> >>> -- >>> Matthieu Dorier >>> ENS Cachan, Brittany (Computer Science dpt.) >>> IRISA Rennes, Office E324 >>> http://perso.eleves.bretagne.ens-cachan.fr/~mdori307/wiki/ >>> >>> _______________________________________________ >>> Pvfs2-users mailing list >>> [email protected] >>> http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users >>> >>> >> > > > -- > Matthieu Dorier > ENS Cachan, Brittany (Computer Science dpt.) > IRISA Rennes, Office E324 > http://perso.eleves.bretagne.ens-cachan.fr/~mdori307/wiki/ > -- Matthieu Dorier ENS Cachan, Brittany (Computer Science dpt.) IRISA Rennes, Office E324 http://perso.eleves.bretagne.ens-cachan.fr/~mdori307/wiki/
_______________________________________________ Pvfs2-users mailing list [email protected] http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users
