xiangyong ouyang wrote: > Hello Kyle, > > > I know it is not possible that ipoib outperforms native IB. So something is > wrong with my testing, but I dont know what's wrong. > > The pvfs2 is compiled using flasg --with-mpi=mpich2-dir, > --with-openib=IB-dir. > > On the clients, the mpiio test program is compiled using mpich2. mpich2 is > built with --enable-romio and -with-pvfs2=/pvfs2-dir. Before testing, clients > mount pvfs2 file system to /tmp/mnt/pvfs2. > > What the mpiio test program does is very simple. mpi processes perform > read/write to a file in pvfs2 file system. > (1) MPI_File_open(comm, opt_file, MPI_MODE_CREATE | MPI_MODE_RDWR, > MPI_INFO_NULL, &fh); // open a file at /tmp/mnt/pvfs2 > (2) MPI_File_seek(fh, seek_position, MPI_SEEK_SET); > (3) MPI_File_write(fh, buf, nchars, MPI_CHAR, &status); > (4) MPI_File_read(fh, buf, nchars, MPI_CHAR, &status); > > > To my surprise, I find that clients can mount pvfs2 without a pvfs2tab file. > Not being a root, I can only put pvfs2tab file in /tmp dir, and I didn't set > the environment PVFS2TAB_FILE to tell pvfs2 client where to find pvfs2tab. > > At clients side using "sudo", I can "insmod pvfs2.ko", and "./pvfs2-client -p > ./pvfs2-client-core". After that, I issue the command "mount -t pvfs2 > ib://$iosrv:3335/pvfs2-fs /tmp/mnt/pvfs2". At this point pvfs2 file system is > mounted. You can view it and operate on it. > > Thus my questions are: > (1) how does a client know where to look for the pvfs2 device without knowing > where to find a pvfs2tab file? > (2) when mpi process access a file in pvfs2 file system, is it talking > directly to pvfs2 servers using IB or tcp/ip or ipoib? How can I verify that? > > There are several default locations that the pvfs2-cp/ls/etc functions use to locate the pvfs2tab:
/etc/fstab /etc/pvfs2tab ./pvfs2tab /etc/mtab If these are not found, then it will not be able to locate the file system properly. In the tabfile you will have specified the protocol to use for connecting to the filesystem: ib://<host>:<port> , etc. When mpi accesses the file, it must also have knowledge of the filesystem tab information so it can look up the correct protocols and paths. If by 'client', you mean how does a process/client know where to look for the device when using the kernel module; that is specified during mounting of the filesystem and I believe the pvfs2-client process keeps track of that information automatically (someone correct me if I'm wrong, you dont need a pvfs2tab file once the client is mounted and running?) You can also gather some very useful information about your filesystem by using the pvfs2-ping admin tool. `pvfs2-ping -m <path to pvfs2 as specified in pvfs2tab. ex: /mnt/pvfs2>` This will list out the information that will be used by your client processes, including mount points, protocols, and location of the tab file being used currently. Hope that helps, ~Kyle > thanks ! > > > > ======= At 2007-07-31, 11:50:46 you wrote: ======= > > >> if you're specifying ib://host:port as your config files, then that >> should be using native IB interfaces. >> >> To see what transport your client is using, you'll need to look at the >> pvfs2tab file and see if it says tcp://host:port (for tcp and ipoib) or >> if it says ib://host:port (for ib) that is what determines the protocol >> to be used by clients. I have yet to see a situation where ipoib >> outperforms the native IB, but these results would be very interesting. >> Can you provide any more information? >> >> Kyle >> >> >> xiangyong ouyang wrote: >> >>> hi, >>> >>> I'm testing pvfs2 over both InfiniBand and IPoIB. For IB, I started pvfs2 >>> server with config file like ib://hostname:3335.... ForIPoIB, pvfs2 >>> servers use config file like tcp://ipoib-address.3334..... >>> >>> Then I mounted pvfs2 file system accrodingly in clients, using either >>> ib://hostname:3335 or tcp://ipoib-address:3334... >>> >>> After that, I run mpi-io test program on clients to measure R/W bandwidth >>> of pvfs2. To my surprise the ipoib gives a better performance number than >>> IB, which can't be true since ipoib involves a lot of overhead on top of >>> IB. >>> >>> So I doubt that clients might not be talking to pvfs2 servers using IB or >>> ipoib as I expect. I want to know what kind of communication transport a >>> client is using (is it tcp/ip, ipoib or IB). How can I implement that task? >>> thanks! >>> >>> >>> xiangyong ouyang >>> >>> _______________________________________________ >>> Pvfs2-users mailing list >>> [email protected] >>> http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users >>> >>> >>> > > = = = = = = = = = = = = = = = = = = = = > > xiangyong ouyang > [EMAIL PROTECTED] > 2007-07-31 > > > _______________________________________________ Pvfs2-users mailing list [email protected] http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users
