I think that test was to used to determine whether the C define macro of HAVE_PVFS2_CREATE_WITHOUT_LAYOUT. In my case, that error leads to defining HAVE_PVFS2_CREATE_WITHOUT_LAYOUT to 1.
FYI. There is no error during the run of make for mpich-3.1.4. The std output of make command is available in http://www.ece.northwestern.edu/~wkliao/mpich-output-make As for OrangeFS, there is no error and the std output of make command is in http://www.ece.northwestern.edu/~wkliao/orangefs-out-make Wei-keng On Mar 30, 2015, at 6:58 PM, Becky Ligon wrote: > Oh! Tomorrow I will check my ROMIO config log and see if I have the same > error. If I don't, then maybe that's why my MPI PVFS2 prefix works and yours > doesn't. > > Did you get any compiler errors when you made OFS? > > Becky > > Sent from my iPhone > >> On Mar 30, 2015, at 7:11 PM, Wei-keng Liao <[email protected]> >> wrote: >> >> >> FYI, from ROMIO's config.log, I am see the following line. >> conftest.c:58: error: too few arguments to function 'PVFS_sys_create' >> >> The config.log file can be found in >> http://www.ece.northwestern.edu/~wkliao/romio-config.log >> >> The log files can be found in >> http://www.ece.northwestern.edu/~wkliao/var-log-messages >> http://www.ece.northwestern.edu/~wkliao/pvfs2-client.log >> http://www.ece.northwestern.edu/~wkliao/pvfs2-server.log >> >> The MPI command I ran is (so you know the file name): >> % mpiexec -n 2 coll_write /orangefs/wkliao/tmpfile.10 >> >> % ls -l /orangefs/wkliao/ >> ls: cannot access /orangefs/wkliao/tmpfile.10: No such file or directory >> total 0 >> ?????????? ? ? ? ? ? tmpfile.10 >> >> Wei-keng >> >>> On Mar 30, 2015, at 5:22 PM, Becky Ligon wrote: >>> >>> 2.6.23 is really old and I have not tried that particular version. >>> >>> Something is awry with your installation, I think. I discovered that when >>> I built mpich-3.1.4, the romio config was looking for pvfs2-config, a >>> program in the OFS bin directory that outputs compile time options. Make >>> sure your mpich installation has the appropriate OFS bin path set in your >>> $PATH and then rebuild your mpich installation...just to be sure. Make >>> sure your OFS installation has the static libraries. Since romio looks for >>> the OFS compile time options, it statically links them into the romio >>> executables. >>> >>> I'm not sure why your version doesn't process the pvfs2 prefix correctly >>> but mine does? Try this on your 2.6.32-504 installation: >>> >>> On your client machine as root, "echo all > /proc/sys/pvfs2/kernel-debug; >>> echo all > /proc/sys/pvfs2/client-debug >>> >>> On the same machine, run your program with the pvfs2 prefix >>> >>> echo none > /proc/sys/pvfs2/kernel-debug; echo none > >>> /proc/sys/pvfs2/client-debug >>> >>> Send me /var/log/messages and your client log file (pvfs2-client.log), so I >>> can see what is/isn't passing through the kernel. >>> >>> Becky >>> >>> On Mon, Mar 30, 2015 at 5:48 PM, Wei-keng Liao >>> <[email protected]> wrote: >>> Hi, Becky >>> >>> Thanks for the update. >>> >>> I have another Linux box that runs an older Linux kernel 2.6.23.9-85.fc8. >>> When I tried your experiment 1 (i.e. ls a non-existent PVFS file, then >>> create the file using pvfs2-touch, and then trying to "ls" the file again) >>> I got this error below. >>> data ::~(4:33pm) #1010 ls /orangefs/wkliao/tmpfile >>> ls: cannot access /orangefs/wkliao/tmpfile: No such file or directory >>> data ::~(4:33pm) #1011 pvfs2-touch /orangefs/wkliao/tmpfile >>> data ::~(4:34pm) #1012 ls /orangefs/wkliao/tmpfile >>> ls: cannot access /orangefs/wkliao/tmpfile: No such file or directory >>> >>> Then I tried your experiment 2 as root. The same error occurs. >>> [root@data wkliao]# echo 3 > /proc/sys/vm/drop_caches >>> [root@data wkliao]# ls /orangefs/wkliao/tmpfile2 >>> ls: cannot access /orangefs/wkliao/tmpfile2: No such file or directory >>> [root@data wkliao]# pvfs2-touch /orangefs/wkliao/tmpfile2 >>> [root@data wkliao]# ls /orangefs/wkliao/tmpfile2 >>> ls: cannot access /orangefs/wkliao/tmpfile2: No such file or directory >>> [root@data wkliao]# cat /proc/sys/vm/drop_caches >>> 3 >>> >>> It looks like the problem also occurs in kernel 2.6.23.9-85.fc8 >>> >>> As for your additional info bullet 3, in my original email, I am >>> having problems with file name prefixed with "pvfs2:" and no prefix. >>> The only one ran correctly is when I used "ufs:" prefix. >>> >>> Wei-keng >>> >>>> On Mar 30, 2015, at 3:48 PM, Becky Ligon wrote: >>>> >>>> Wei-keng: >>>> >>>> Just wanted to give you an update: I have discovered that newer versions >>>> of the kernel, i.e., 3.10.0-123.20.1.el7.x86_64 distributed via CentOS 7.0 >>>> do not exhibit the problem that is found using kernel >>>> 2.6.32-504.12.2.el6.x86_64 distributed by Scientific Linux 6.6. As I >>>> mentioned before, kernel 2.6.32-358.14.1.el6.x86_64 distributed by >>>> Scientific Linux 6.4 didn't exhibit the problem behavior either. >>>> >>>> When I run your program coll_write without prefacing the >>>> /path/to/filename, mpi issues a lstat through the kernel, which passes the >>>> request to PVFS. PVFS returns a "file not found" and marks that filename >>>> as an invalid inode and stores this information in the kernel's dcache >>>> (directory cache). mpi then uses the PVFS libraries to complete the >>>> creation of the file. The libraries bypass the kernel, so the kernel >>>> doesn't know that this file has been created. Since the inode has been >>>> marked invalid in the kernel, the "ls" command sees this file as >>>> "file-not-found". This kernel behavior is wrong. The "ls" *SHOULD* be >>>> passed to the PVFS kernel module and on to the client core where a request >>>> for this file can be sent to the servers. Since I can't change the >>>> behavior of the kernel, I am experimenting to see if we need to mark the >>>> inode invalid in the first place. This is where my efforts have gotten me >>>> so far. >>>> >>>> Additional information: >>>> >>>> 1. I can reproduce this problem by issuing an "ls" on a non-existent PVFS >>>> file, then create the file using pvfs2-touch, and then trying to "ls" the >>>> file again. >>>> >>>> 2. If you drop the dcache ( echo 3 > /proc/sys/vm/drop_caches), you can >>>> "ls" the newly created file with no problem. >>>> >>>> 3. With your coll_write program, if you preface the file with pvfs2 or >>>> ufs ({pvfs2|ufs}:/path/to/filename), the system behaves properly. When >>>> the pvfs2 prefix is used, nothing goes through the kernel; everything goes >>>> through the libraries. When the ufs prefix is used, everything goes >>>> through the kernel and on to PVFS. >>>> >>>> >>>> Becky >>>> >>>> On Tue, Mar 17, 2015 at 6:10 PM, Becky Ligon <[email protected]> wrote: >>>> I'm heading out for the evening but will jump on the problem tomorrow! >>>> >>>> Thanks for finding this problem! >>>> >>>> Becky >>>> >>>> On Tue, Mar 17, 2015 at 6:07 PM, Wei-keng Liao >>>> <[email protected]> wrote: >>>> Alright! It is good to know. >>>> I will stop messing around my builds and wait for your good news. >>>> Thanks. >>>> >>>> Wei-keng >>>> >>>>> On Mar 17, 2015, at 5:05 PM, Becky Ligon wrote: >>>>> >>>>> [bligon@SL6 wkliao]$ ls -l /mnt/test/wkliao >>>>> ls: cannot access /mnt/test/wkliao/testfile: No such file or directory >>>>> total 0 >>>>> ?????????? ? ? ? ? ? testfile >>>>> >>>>> >>>>> On Tue, Mar 17, 2015 at 6:04 PM, Becky Ligon <[email protected]> wrote: >>>>> Okay. I just recreated your problem. There is something amuck with the >>>>> newer kernel. Let me work on it and get back to you! >>>>> >>>>> Becky >>>>> >>>>> On Tue, Mar 17, 2015 at 6:01 PM, Wei-keng Liao >>>>> <[email protected]> wrote: >>>>> % cat /etc/redhat-release >>>>> Red Hat Enterprise Linux Server release 6.6 (Santiago) >>>>> >>>>> >>>>> Wei-keng >>>>> >>>>>> On Mar 17, 2015, at 5:01 PM, Becky Ligon wrote: >>>>>> >>>>>> issue: cat /etc/redhat-release >>>>>> >>>>>> On Tue, Mar 17, 2015 at 6:00 PM, Becky Ligon <[email protected]> wrote: >>>>>> Yes, but are you running CentOS, SL, ????? >>>>>> >>>>>> On Tue, Mar 17, 2015 at 5:54 PM, Wei-keng Liao >>>>>> <[email protected]> wrote: >>>>>> The command uname -a shows >>>>>> >>>>>> Linux bigdata.eecs.northwestern.edu 2.6.32-504.8.1.el6.x86_64 #1 SMP Fri >>>>>> Dec 19 12:09:25 EST 2014 x86_64 x86_64 x86_64 GNU/Linux >>>>>> >>>>>> Wei-keng >>>>>> >>>>>>> On Mar 17, 2015, at 4:52 PM, Becky Ligon wrote: >>>>>>> >>>>>>> Good! >>>>>>> >>>>>>> I'm working on the getting the system up and running with the newer >>>>>>> kernel. >>>>>>> >>>>>>> Which distro are you using? >>>>>>> >>>>>>> Becky >>>>>>> >>>>>>> On Tue, Mar 17, 2015 at 5:51 PM, Wei-keng Liao >>>>>>> <[email protected]> wrote: >>>>>>> Versioning is not an issue, as my older version of orangefs is on a >>>>>>> different machine. >>>>>>> This machine is a fresh install for Orangefs, BerkeleyDB, and MPICH. >>>>>>> >>>>>>> Wei-keng >>>>>>> >>>>>>>> On Mar 17, 2015, at 4:48 PM, Becky Ligon wrote: >>>>>>>> >>>>>>>> Could it be that you have a versioning issue here? Somehow, you have >>>>>>>> a mix of 2.9.1 and some older version? >>>>>>>> >>>>>>>> I was told by our tester that he always tests with --enabled-shared. >>>>>>>> I will try it both ways. Maybe, that's not the case! >>>>>>>> >>>>>>>> Becky >>>>>>>> >>>>>>>> On Tue, Mar 17, 2015 at 5:43 PM, Wei-keng Liao >>>>>>>> <[email protected]> wrote: >>>>>>>> Here is from my /var/log/messages when I restart the pvfs2 >>>>>>>> server/client. >>>>>>>> >>>>>>>> Mar 16 12:20:50 bigdata kernel: pvfs2: module version 2.9.1- unloaded >>>>>>>> Mar 16 12:20:53 bigdata kernel: pvfs2: pvfs2_init called with debug >>>>>>>> mask: "none" (0x00000000) >>>>>>>> Mar 16 12:20:53 bigdata kernel: pvfs2: module version 2.9.1- loaded >>>>>>>> Mar 16 12:20:55 bigdata kernel: PVFS: kernel debug mask has been >>>>>>>> modified to "none" (0x00000000) >>>>>>>> Mar 16 12:20:55 bigdata kernel: PVFS: client debug mask has been >>>>>>>> modified to "none" (0x00000000) >>>>>>>> >>>>>>>> Are you saying that MPICH required PVFS2 built with --enable-shared >>>>>>>> option? >>>>>>>> I don't know about this, as I had an older version of pvfs2 running >>>>>>>> fine and it was >>>>>>>> not built with that option. >>>>>>>> >>>>>>>> Rob, do you know? >>>>>>>> >>>>>>>> >>>>>>>> Wei-keng >>>>>>>> >>>>>>>>> On Mar 17, 2015, at 4:35 PM, Becky Ligon wrote: >>>>>>>>> >>>>>>>>> I'm also updating my kernel to 2.6.32-504.12.2.el6 and will rerun my >>>>>>>>> previous tests to see if the kernel is the problem. >>>>>>>>> >>>>>>>>> In /var/log/messages, you should also see a message like: >>>>>>>>> >>>>>>>>> Mar 17 15:15:42 SL6 kernel: pvfs2: module version 2.9.1- loaded >>>>>>>>> >>>>>>>>> If you are not seeing this message, then it appears the kernel module >>>>>>>>> did not get loaded. >>>>>>>>> >>>>>>>>> My understanding is that MPI requires the shared libraries. >>>>>>>>> >>>>>>>>> Becky >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On Tue, Mar 17, 2015 at 5:27 PM, Wei-keng Liao >>>>>>>>> <[email protected]> wrote: >>>>>>>>> I tried to build pvfs2 with --enable-shared before and the same >>>>>>>>> occurred. >>>>>>>>> >>>>>>>>> I am going to try install orangefs on one computer only and not using >>>>>>>>> any symbolic link. >>>>>>>>> will let you know. >>>>>>>>> >>>>>>>>> I see the following from the pvfs2 client log file. >>>>>>>>> [D 03/16/2015 12:20:55] [INFO]: Mapping pointer 0x7ff8c0087000 for >>>>>>>>> I/O. >>>>>>>>> [D 03/16/2015 12:20:55] [INFO]: Mapping pointer 0x197e000 for I/O. >>>>>>>>> >>>>>>>>> I see the following from /var/log/message >>>>>>>>> Mar 16 10:19:35 bigdata kernel: pvfs2: module version 2.9.1- unloaded >>>>>>>>> Mar 16 10:19:37 bigdata kernel: pvfs2: pvfs2_init called with debug >>>>>>>>> mask: "none" (0x00000000) >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> Wei-keng >>>>>>>>> >>>>>>>>>> On Mar 17, 2015, at 4:04 PM, Becky Ligon wrote: >>>>>>>>>> >>>>>>>>>> Try removing the symbolic link. Shut everything down; recreate your >>>>>>>>>> storage, and restart. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> If that doesn't work, then I want you to rebuild the software using >>>>>>>>>> --enable-shared as one of your configure options. Recreate >>>>>>>>>> everything, including your storage. Before you start the client, >>>>>>>>>> you will have to add LD_LIBRARY_PATH to your environment and point >>>>>>>>>> it to the OFS installation lib directory. Recompile your mpi >>>>>>>>>> programs making sure you have the following set: >>>>>>>>>> >>>>>>>>>> C_INCLUDE_PATH contains the include directory for OFS and the >>>>>>>>>> LIBRARY_PATH includes the OFS lib directory. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Are you seeing any errors in the pvfs2-client.log file or >>>>>>>>>> /var/log/messages? >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> If none of the above works, then we can turn on kernel debugging >>>>>>>>>> and/or client debugging to see where the problem is coming from. >>>>>>>>>> >>>>>>>>>> Let me know! >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Becky >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Tue, Mar 17, 2015 at 4:44 PM, Wei-keng Liao >>>>>>>>>> <[email protected]> wrote: >>>>>>>>>> >>>>>>>>>> The command "mount" shows /home directory is not nfs and there is no >>>>>>>>>> nfs client running on this machine (the metadata server). >>>>>>>>>>>>> /dev/sda7 on /home type ext4 (rw,usrquota) >>>>>>>>>> >>>>>>>>>> The command "df" also shows /home is not NSF mounted. >>>>>>>>>> bigdata::~(3:21pm) #1003 df >>>>>>>>>> Filesystem 1K-blocks Used Available Use% Mounted on >>>>>>>>>> /dev/sda2 30106576 8053816 20516760 29% / >>>>>>>>>> tmpfs 16414648 0 16414648 0% /dev/shm >>>>>>>>>> /dev/sda1 3997376 90920 3696744 3% /boot >>>>>>>>>> /dev/sda7 850331204 373303148 433827084 47% /home >>>>>>>>>> /dev/sda3 50264772 53104 47651668 1% /tmp >>>>>>>>>> /dev/sda6 9948012 875016 8560996 10% /var >>>>>>>>>> tcp://bigdata:3334/orangefs >>>>>>>>>> 3228520448 1581383680 1647136768 49% /orangefs >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> My guess it is still the kernel module that does not behave >>>>>>>>>> correctly. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Wei-keng >>>>>>>>>> >>>>>>>>>>> On Mar 17, 2015, at 3:26 PM, Becky Ligon wrote: >>>>>>>>>>> >>>>>>>>>>> I also meant to say that if your storage is on a nfs mount, move >>>>>>>>>>> your filesystem onto local storage, no links or nfs mounts. >>>>>>>>>>> >>>>>>>>>>> Becky >>>>>>>>>>> >>>>>>>>>>> On Tue, Mar 17, 2015 at 4:14 PM, Becky Ligon <[email protected]> >>>>>>>>>>> wrote: >>>>>>>>>>> [bligon@SL6 wkliao]$ ls -l /mnt/test/wkliao >>>>>>>>>>> total 91804 >>>>>>>>>>> -rw-r--r-- 1 bligon bligon 31000000 Mar 17 16:06 testfile >>>>>>>>>>> -rw-r--r-- 1 bligon bligon 31000000 Mar 17 16:06 testfile.pvfs2 >>>>>>>>>>> -rw-rw-rw- 1 bligon bligon 32000000 Mar 17 16:05 testfile.ufs >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> [bligon@SL6 wkliao]$ cat /proc/sys/pvfs2/acache/timeout-msecs >>>>>>>>>>> 60000 >>>>>>>>>>> [bligon@SL6 wkliao]$ cat /proc/sys/pvfs2/ncache/timeout-msecs >>>>>>>>>>> 60000 >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> I was able to run your programs and issue an "ls" with no problems, >>>>>>>>>>> even with timeouts set at 60 seconds. My environment is slightly >>>>>>>>>>> different: my distro is 2.6.32-358.14.1.el6.x86_64. >>>>>>>>>>> >>>>>>>>>>> Is your storage on a NFS-mounted directory? You said in an earlier >>>>>>>>>>> post that /files1 was a symbolic link to /home. Does /home reside >>>>>>>>>>> on a NFS mounted filesystem? If not, then try recreating your >>>>>>>>>>> storage without the symbolic link. >>>>>>>>>>> >>>>>>>>>>> Becky >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Tue, Mar 17, 2015 at 4:00 PM, Becky Ligon <[email protected]> >>>>>>>>>>> wrote: >>>>>>>>>>> Got it. Thanks. >>>>>>>>>>> >>>>>>>>>>> I've installed mpich 3.1.4 on my local vm and have created a >>>>>>>>>>> filesystem just like yours. Now, I'm going to try running your >>>>>>>>>>> program and see what shakes out! >>>>>>>>>>> >>>>>>>>>>> Becky >>>>>>>>>>> >>>>>>>>>>> On Tue, Mar 17, 2015 at 3:42 PM, Wei-keng Liao >>>>>>>>>>> <[email protected]> wrote: >>>>>>>>>>> Sorry, please try again. >>>>>>>>>>> >>>>>>>>>>> Wei-keng >>>>>>>>>>> >>>>>>>>>>>> On Mar 17, 2015, at 2:22 PM, Becky Ligon wrote: >>>>>>>>>>>> >>>>>>>>>>>> Wei-keng: >>>>>>>>>>>> >>>>>>>>>>>> I can't access coll_write.c. Permission denied. >>>>>>>>>>>> >>>>>>>>>>>> Can you grant me the appropriate permissions so I can copy the >>>>>>>>>>>> code? >>>>>>>>>>>> >>>>>>>>>>>> Thanks! >>>>>>>>>>>> Becky >>>>>>>>>>>> >>>>>>>>>>>> On Mon, Mar 16, 2015 at 5:31 PM, Becky Ligon <[email protected]> >>>>>>>>>>>> wrote: >>>>>>>>>>>> Thanks! Still working on it! >>>>>>>>>>>> >>>>>>>>>>>> Becky >>>>>>>>>>>> >>>>>>>>>>>>> On Mon, Mar 16, 2015 at 5:07 PM, Wei-keng Liao >>>>>>>>>>>>> <[email protected]> wrote: >>>>>>>>>>>>> How did you compile your programs? >>>>>>>>>>>> >>>>>>>>>>>> My test program can be found in >>>>>>>>>>>> http://www.ece.northwestern.edu/~wkliao/coll_write.c >>>>>>>>>>>> >>>>>>>>>>>> bigdata::~/TEST_PROG(3:57pm) #1053 make coll_write >>>>>>>>>>>> mpicc -g -o coll_write coll_write.c >>>>>>>>>>>> >>>>>>>>>>>> The command I ran: >>>>>>>>>>>> mpiexec -n 2 coll_write /orangefs/wkliao/testfile >>>>>>>>>>>> >>>>>>>>>>>> bigdata::~/TEST_PROG(3:58pm) #1055 mpicc --version >>>>>>>>>>>> gcc (GCC) 4.4.7 20120313 (Red Hat 4.4.7-11) >>>>>>>>>>>> Copyright (C) 2010 Free Software Foundation, Inc. >>>>>>>>>>>> This is free software; see the source for copying conditions. >>>>>>>>>>>> There is NO >>>>>>>>>>>> warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR >>>>>>>>>>>> PURPOSE. >>>>>>>>>>>> >>>>>>>>>>>> bigdata::~/TEST_PROG(3:58pm) #1056 mpichversion >>>>>>>>>>>> MPICH Version: 3.1.4 >>>>>>>>>>>> MPICH Release date: Fri Feb 20 15:02:56 CST 2015 >>>>>>>>>>>> MPICH Device: ch3:nemesis >>>>>>>>>>>> MPICH configure: --enable-g=debug --disable-fast >>>>>>>>>>>> --enable-shared --enable-fc --enable-cxx --enable-romio >>>>>>>>>>>> --with-file-system=ufs+pvfs2 CC=gcc CXX=g++ FC=gfortran CFLAGS=-g >>>>>>>>>>>> -O0 FCFLAGS=-g -O0 >>>>>>>>>>>> MPICH CC: gcc -g -O0 -g -O0 >>>>>>>>>>>> MPICH CXX: g++ -g -O0 >>>>>>>>>>>> MPICH F77: gfortran -g -O0 >>>>>>>>>>>> MPICH FC: gfortran -g -O0 -g -O0 >>>>>>>>>>>> >>>>>>>>>>>>> Can you send me a listing of the /files1 directories where your >>>>>>>>>>>>> storage is located? >>>>>>>>>>>> >>>>>>>>>>>> /files1 is a symbolic link to the local disk mounted at /home >>>>>>>>>>>> >>>>>>>>>>>> bigdata::~(3:55pm) #1043 ls -l /files1 >>>>>>>>>>>> total 0 >>>>>>>>>>>> lrwxrwxrwx 1 root root 21 Mar 12 09:15 orangefs -> >>>>>>>>>>>> /home/files1/orangefs >>>>>>>>>>>> >>>>>>>>>>>> bigdata::~(3:55pm) #1044 ldd /usr/local/sbin/pvfs2-client >>>>>>>>>>>> linux-vdso.so.1 => (0x00007fff54db6000) >>>>>>>>>>>> librt.so.1 => /lib64/librt.so.1 (0x0000003fc2400000) >>>>>>>>>>>> libm.so.6 => /lib64/libm.so.6 (0x0000003fc1800000) >>>>>>>>>>>> libdl.so.2 => /lib64/libdl.so.2 (0x0000003fc2000000) >>>>>>>>>>>> libpthread.so.0 => /lib64/libpthread.so.0 >>>>>>>>>>>> (0x0000003fc1c00000) >>>>>>>>>>>> libssl.so.10 => /usr/lib64/libssl.so.10 (0x0000003fcb000000) >>>>>>>>>>>> libcrypto.so.10 => /usr/lib64/libcrypto.so.10 >>>>>>>>>>>> (0x0000003fc8400000) >>>>>>>>>>>> libc.so.6 => /lib64/libc.so.6 (0x0000003fc1400000) >>>>>>>>>>>> /lib64/ld-linux-x86-64.so.2 (0x0000003fc1000000) >>>>>>>>>>>> libgssapi_krb5.so.2 => /lib64/libgssapi_krb5.so.2 >>>>>>>>>>>> (0x0000003fca800000) >>>>>>>>>>>> libkrb5.so.3 => /lib64/libkrb5.so.3 (0x0000003fcac00000) >>>>>>>>>>>> libcom_err.so.2 => /lib64/libcom_err.so.2 >>>>>>>>>>>> (0x0000003fc8c00000) >>>>>>>>>>>> libk5crypto.so.3 => /lib64/libk5crypto.so.3 >>>>>>>>>>>> (0x0000003fca400000) >>>>>>>>>>>> libz.so.1 => /lib64/libz.so.1 (0x0000003fc2800000) >>>>>>>>>>>> libkrb5support.so.0 => /lib64/libkrb5support.so.0 >>>>>>>>>>>> (0x0000003fc9800000) >>>>>>>>>>>> libkeyutils.so.1 => /lib64/libkeyutils.so.1 >>>>>>>>>>>> (0x0000003fc9000000) >>>>>>>>>>>> libresolv.so.2 => /lib64/libresolv.so.2 (0x0000003fc3400000) >>>>>>>>>>>> libselinux.so.1 => /lib64/libselinux.so.1 >>>>>>>>>>>> (0x0000003fc3000000) >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Wei-keng >>>>>>>>>>>> >>>>>>>>>>>>> On Mar 16, 2015, at 3:49 PM, Becky Ligon wrote: >>>>>>>>>>>>> >>>>>>>>>>>>> Also: >>>>>>>>>>>>> >>>>>>>>>>>>> ldd ./pvfs2-client >>>>>>>>>>>>> >>>>>>>>>>>>> Becky >>>>>>>>>>>>> >>>>>>>>>>>>> On Mon, Mar 16, 2015 at 4:47 PM, Becky Ligon <[email protected]> >>>>>>>>>>>>> wrote: >>>>>>>>>>>>> Thanks! >>>>>>>>>>>>> >>>>>>>>>>>>> How did you compile your programs? >>>>>>>>>>>>> >>>>>>>>>>>>> Just checking everything. So far, all of your settings look good. >>>>>>>>>>>>> >>>>>>>>>>>>> Can you send me a listing of the /files1 directories where your >>>>>>>>>>>>> storage is located? >>>>>>>>>>>>> >>>>>>>>>>>>> Becky >>>>>>>>>>>>> >>>>>>>>>>>>> On Mon, Mar 16, 2015 at 4:32 PM, Wei-keng Liao >>>>>>>>>>>>> <[email protected]> wrote: >>>>>>>>>>>>> Here are the outputs. >>>>>>>>>>>>> >>>>>>>>>>>>> bigdata::~(3:27pm) #1030 /sbin/lsmod | grep pvfs >>>>>>>>>>>>> pvfs2 139546 2 >>>>>>>>>>>>> >>>>>>>>>>>>> bigdata::~(3:27pm) #1031 ps aux | grep pvfs >>>>>>>>>>>>> root 5775 0.5 0.0 321788 11228 ? Ssl 12:20 1:02 >>>>>>>>>>>>> /usr/local/sbin/pvfs2-server --pidfile /var/run/pvfs2.pid >>>>>>>>>>>>> /etc/orangefs-server.conf >>>>>>>>>>>>> root 5791 0.0 0.0 39640 472 ? Ss 12:20 0:00 >>>>>>>>>>>>> /usr/local/sbin/pvfs2-client -p /usr/local/sbin/pvfs2-client-core >>>>>>>>>>>>> --logfile /files1/orangefs/client.log --logstamp=datetime -a 0 -n >>>>>>>>>>>>> 0 >>>>>>>>>>>>> root 5792 0.2 0.0 79904 27532 ? SL 12:20 0:25 >>>>>>>>>>>>> pvfs2-client-core --child -a 0 -n 0 --logtype file -L >>>>>>>>>>>>> /files1/orangefs/client.log --logstamp datetime >>>>>>>>>>>>> wkliao 7037 0.0 0.0 105316 864 pts/3 S+ 15:27 0:00 >>>>>>>>>>>>> grep pvfs >>>>>>>>>>>>> >>>>>>>>>>>>> bigdata::~(3:27pm) #1032 cat /etc/pvfs2tab >>>>>>>>>>>>> tcp://bigdata:3334/orangefs /orangefs pvfs2 default,noauto 0 0 >>>>>>>>>>>>> >>>>>>>>>>>>> bigdata::~(3:27pm) #1033 /bin/mount >>>>>>>>>>>>> /dev/sda2 on / type ext4 (rw) >>>>>>>>>>>>> proc on /proc type proc (rw) >>>>>>>>>>>>> sysfs on /sys type sysfs (rw) >>>>>>>>>>>>> devpts on /dev/pts type devpts (rw,gid=5,mode=620) >>>>>>>>>>>>> tmpfs on /dev/shm type tmpfs (rw) >>>>>>>>>>>>> /dev/sda1 on /boot type ext4 (rw) >>>>>>>>>>>>> /dev/sda7 on /home type ext4 (rw,usrquota) >>>>>>>>>>>>> /dev/sda3 on /tmp type ext4 (rw) >>>>>>>>>>>>> /dev/sda6 on /var type ext4 (rw) >>>>>>>>>>>>> none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw) >>>>>>>>>>>>> sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw) >>>>>>>>>>>>> nfsd on /proc/fs/nfsd type nfsd (rw) >>>>>>>>>>>>> tcp://bigdata:3334/orangefs on /orangefs type pvfs2 (rw) >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Wei-keng >>>>>>>>>>>>> >>>>>>>>>>>>>> On Mar 16, 2015, at 3:21 PM, Becky Ligon wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>> Can you also issue: >>>>>>>>>>>>>> >>>>>>>>>>>>>> /bin/mount >>>>>>>>>>>>>> >>>>>>>>>>>>>> so, I can see if the filesystem is mounted correctly? >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Mon, Mar 16, 2015 at 4:13 PM, Becky Ligon <[email protected]> >>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>> On your system where you have the client and server running, can >>>>>>>>>>>>>> you send me the output from the following commands: >>>>>>>>>>>>>> >>>>>>>>>>>>>> /sbin/lsmod | grep pvfs >>>>>>>>>>>>>> ps aux | grep pvfs >>>>>>>>>>>>>> cat /etc/pvfs2tab >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Mon, Mar 16, 2015 at 4:01 PM, Becky Ligon <[email protected]> >>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>> I'm running a test case to see if I can reproduce your problem. >>>>>>>>>>>>>> I'll get back with you shortly. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Becky >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Mon, Mar 16, 2015 at 3:40 PM, Wei-keng Liao >>>>>>>>>>>>>> <[email protected]> wrote: >>>>>>>>>>>>>> Here you go. >>>>>>>>>>>>>> >>>>>>>>>>>>>> http://www.ece.northwestern.edu/~wkliao/orangefs-server.conf >>>>>>>>>>>>>> >>>>>>>>>>>>>> Wei-keng >>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Mar 16, 2015, at 2:13 PM, Becky Ligon wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> PVFS2_SERVER_CONF=/etc/orangefs-server.conf >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Mon, Mar 16, 2015 at 3:10 PM, Wei-keng Liao >>>>>>>>>>>>>>> <[email protected]> wrote: >>>>>>>>>>>>>>> I am not sure what file that is? >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Wei-keng >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On Mar 16, 2015, at 2:04 PM, Becky Ligon wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Now send me a copy of your server config file. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On Mon, Mar 16, 2015 at 2:37 PM, Wei-keng Liao >>>>>>>>>>>>>>>> <[email protected]> wrote: >>>>>>>>>>>>>>>> The requested 2 files can be found in the following URLs. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> http://www.ece.northwestern.edu/~wkliao/config.log >>>>>>>>>>>>>>>> http://www.ece.northwestern.edu/~wkliao/pvfs2-server >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Wei-keng >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> On Mar 16, 2015, at 1:29 PM, Becky Ligon wrote: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Can you also send me your config.log file from when you >>>>>>>>>>>>>>>>> compiled the source? >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Becky >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> On Mon, Mar 16, 2015 at 2:28 PM, Becky Ligon >>>>>>>>>>>>>>>>> <[email protected]> wrote: >>>>>>>>>>>>>>>>> If you are experimenting with OrangeFS, then having one >>>>>>>>>>>>>>>>> metadata and 4 data servers is fine. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Can you send me your pvfs2-server init file, the one used >>>>>>>>>>>>>>>>> with the /sbin/service command? >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Becky >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> On Mon, Mar 16, 2015 at 1:27 PM, Wei-keng Liao >>>>>>>>>>>>>>>>> <[email protected]> wrote: >>>>>>>>>>>>>>>>> On Mar 16, 2015, at 12:02 PM, Becky Ligon wrote: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Wei-Keing: >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Did you umount and mount the filesystem? If not, umount the >>>>>>>>>>>>>>>>>> filesystem, restart the client core, and then mount the >>>>>>>>>>>>>>>>>> filesystem again. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Yes. My restart command ran "/sbin/servive pvfs2-server >>>>>>>>>>>>>>>>> restart" >>>>>>>>>>>>>>>>> the script contains both client and server commands, include >>>>>>>>>>>>>>>>> client's umount and mount. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> I also suggest that you define your environment so that each >>>>>>>>>>>>>>>>>> of your machines (bigdata, bigdata1, bigdata2, bigdata3) >>>>>>>>>>>>>>>>>> have their pvfs servers configured to handle both I/O and >>>>>>>>>>>>>>>>>> metadata. To do this, you will have to recreate the >>>>>>>>>>>>>>>>>> filesystem. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> My Orangefs is newly created and the test program is the >>>>>>>>>>>>>>>>> first one run in parallel on it. >>>>>>>>>>>>>>>>> Isn't my configuration legit (one metadata sever and 4 data >>>>>>>>>>>>>>>>> servers) for orangefs setup? >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Again, my mpi test program ran 2 processes locally on the >>>>>>>>>>>>>>>>> metadata server which is both data server and client. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Wei-keng >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Becky >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> On Mon, Mar 16, 2015 at 11:30 AM, Wei-keng Liao >>>>>>>>>>>>>>>>>> <[email protected]> wrote: >>>>>>>>>>>>>>>>>> HI, Becky >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> I tried the command option "-a 0 -n 0" and restart the >>>>>>>>>>>>>>>>>> client/server, but the same issue persists. >>>>>>>>>>>>>>>>>> pvfs2-ping command shows one metadata server and 4 data >>>>>>>>>>>>>>>>>> servers. >>>>>>>>>>>>>>>>>> I ran my test program on the metadata server. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> meta servers: >>>>>>>>>>>>>>>>>> tcp://bigdata:3334 Ok >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> data servers: >>>>>>>>>>>>>>>>>> tcp://bigdata:3334 Ok >>>>>>>>>>>>>>>>>> tcp://bigdata1:3334 Ok >>>>>>>>>>>>>>>>>> tcp://bigdata2:3334 Ok >>>>>>>>>>>>>>>>>> tcp://bigdata3:3334 Ok >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Wei-keng >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> On Mar 16, 2015, at 8:26 AM, Becky Ligon wrote: >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Caching is still an issue if you have servers on more than >>>>>>>>>>>>>>>>>>> one machine and those servers provide metadata. Even in a >>>>>>>>>>>>>>>>>>> one-server environment, it could make a difference. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> The "ls" command uses the kernel module and client core, >>>>>>>>>>>>>>>>>>> which in turn use the caches, while the pvfs2-ls command >>>>>>>>>>>>>>>>>>> does not. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> If you don't have the appropriate sudo permissions to >>>>>>>>>>>>>>>>>>> modify the /proc filesystem, then you can start the client >>>>>>>>>>>>>>>>>>> with the caches turned off. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Example: >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> ./pvfs2-client -a 0 -n 0 >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> If you execute pvfs2-client --help, you will see these >>>>>>>>>>>>>>>>>>> options. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Becky >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Sent from my iPhone >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> On Mar 15, 2015, at 5:05 PM, Wei-keng Liao >>>>>>>>>>>>>>>>>>>> <[email protected]> wrote: >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> I assume after 60 seconds, the client will flush the cache. >>>>>>>>>>>>>>>>>>>> Please note I am running orangefs client and server on the >>>>>>>>>>>>>>>>>>>> same machine. >>>>>>>>>>>>>>>>>>>> In this case, should caching become an issue? >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Long after 60 seconds of the file creation, command ls >>>>>>>>>>>>>>>>>>>> still could not find the file. >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> I got permission denied when running the two echo commands >>>>>>>>>>>>>>>>>>>> you suggested. >>>>>>>>>>>>>>>>>>>> I DO have sudo permission. I also tried vi those files but >>>>>>>>>>>>>>>>>>>> got error of >>>>>>>>>>>>>>>>>>>> "/proc/sys/pvfs2/acache/timeout-msecs" E667: Fsync failed >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Also, how do I set this automatically after system reboot? >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Wei-keng >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> On Mar 15, 2015, at 2:16 PM, Becky Ligon wrote: >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> Wei-keng: >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> This is most likely a caching issue with the client. By >>>>>>>>>>>>>>>>>>>>> default, we set the cache to timeout after 60 seconds, >>>>>>>>>>>>>>>>>>>>> which may be too high in your environment. Or, you have >>>>>>>>>>>>>>>>>>>>> deleted and redefined a file using the same name outside >>>>>>>>>>>>>>>>>>>>> of the client where you are seeing the question marks, in >>>>>>>>>>>>>>>>>>>>> which case, the cache would be wrong for that file. >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> To verify, turn off caching to see if this resolves your >>>>>>>>>>>>>>>>>>>>> problem: >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> As root on your client machine: >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> echo "0" > /proc/sys/pvfs2/acache/timeout-msecs >>>>>>>>>>>>>>>>>>>>> echo "0" > /proc/sys/pvfs2/ncache/timeout-msecs >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> If this change fixes your problem, try setting the >>>>>>>>>>>>>>>>>>>>> timeout-msecs to something more appropriate for your >>>>>>>>>>>>>>>>>>>>> environment. >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> Becky >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> On Sun, Mar 15, 2015 at 11:43 AM, Wei-keng Liao >>>>>>>>>>>>>>>>>>>>> <[email protected]> wrote: >>>>>>>>>>>>>>>>>>>>> Hi >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> I am having problems with OrangeFS 2.9.1 and MPICH 3.1.4. >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> Here is my system settings: >>>>>>>>>>>>>>>>>>>>> Linux Kernel 2.6.32 >>>>>>>>>>>>>>>>>>>>> Berkeley DB version 6.1.19 >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> A simple MPI test program that calls MPI_File_open and >>>>>>>>>>>>>>>>>>>>> MPI_File_write_all >>>>>>>>>>>>>>>>>>>>> is used and ran two processes on the same host. >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> The MPI commands I used with prefix file names to force >>>>>>>>>>>>>>>>>>>>> ADIO drivers: >>>>>>>>>>>>>>>>>>>>> mpiexec -n 2 coll_write /orangefs/wkliao/testfile >>>>>>>>>>>>>>>>>>>>> mpiexec -n 2 coll_write >>>>>>>>>>>>>>>>>>>>> pvfs2:/orangefs/wkliao/testfile.pvfs2 >>>>>>>>>>>>>>>>>>>>> mpiexec -n 2 coll_write ufs:/orangefs/wkliao/testfile.ufs >>>>>>>>>>>>>>>>>>>>> The first two will use the pvfs2 driver and the 3rd the >>>>>>>>>>>>>>>>>>>>> ufs driver. >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> Here is what I see when running "ls -l" and "pvfs2-ls -l" >>>>>>>>>>>>>>>>>>>>> commands. >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> % ls -l /orangefs/wkliao/ >>>>>>>>>>>>>>>>>>>>> ls: cannot access /orangefs/wkliao/testfile: No such file >>>>>>>>>>>>>>>>>>>>> or directory >>>>>>>>>>>>>>>>>>>>> ls: cannot access /orangefs/wkliao/testfile.pvfs2: No >>>>>>>>>>>>>>>>>>>>> such file or directory >>>>>>>>>>>>>>>>>>>>> total 31252 >>>>>>>>>>>>>>>>>>>>> ?????????? ? ? ? ? ? testfile >>>>>>>>>>>>>>>>>>>>> ?????????? ? ? ? ? ? >>>>>>>>>>>>>>>>>>>>> testfile.pvfs2 >>>>>>>>>>>>>>>>>>>>> -rw------- 1 wkliao users 32000000 Mar 13 18:55 >>>>>>>>>>>>>>>>>>>>> testfile.ufs >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> % pvfs2-ls -l /orangefs/wkliao/ >>>>>>>>>>>>>>>>>>>>> -rw-r--r-- 1 wkliao users 31000000 2015-03-13 >>>>>>>>>>>>>>>>>>>>> 18:55 testfile >>>>>>>>>>>>>>>>>>>>> -rw------- 1 wkliao users 32000000 2015-03-13 >>>>>>>>>>>>>>>>>>>>> 18:55 testfile.ufs >>>>>>>>>>>>>>>>>>>>> -rw-r--r-- 1 wkliao users 31000000 2015-03-13 >>>>>>>>>>>>>>>>>>>>> 18:55 testfile.pvfs2 >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> My config.log file for building >>>>>>>>>>>>>>>>>>>>> orangefhttp://www.scientificlinux.org/downloads/sl-versions/sl6/s >>>>>>>>>>>>>>>>>>>>> can be found in this URL >>>>>>>>>>>>>>>>>>>>> http://www.ece.northwestern.edu/~wkliao/config.log >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> Wei-keng >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>>>>>>>>> Pvfs2-users mailing list >>>>>>>>>>>>>>>>>>>>> [email protected] >>>>>>>>>>>>>>>>>>>>> http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>> Becky Ligon >>>>>>>>>>>>>>>>>> Research Associate >>>>>>>>>>>>>>>>>> Clemson University >>>>>>>>>>>>>>>>>> Clemson, SC >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>> Becky Ligon >>>>>>>>>>>>>>>>> Research Associate >>>>>>>>>>>>>>>>> Clemson University >>>>>>>>>>>>>>>>> Clemson, SC >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>> Becky Ligon >>>>>>>>>>>>>>>>> Research Associate >>>>>>>>>>>>>>>>> Clemson University >>>>>>>>>>>>>>>>> Clemson, SC >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>> Becky Ligon >>>>>>>>>>>>>>>> Research Associate >>>>>>>>>>>>>>>> Clemson University >>>>>>>>>>>>>>>> Clemson, SC >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>> Becky Ligon >>>>>>>>>>>>>>> Research Associate >>>>>>>>>>>>>>> Clemson University >>>>>>>>>>>>>>> Clemson, SC >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> -- >>>>>>>>>>>>>> Becky Ligon >>>>>>>>>>>>>> Research Associate >>>>>>>>>>>>>> Clemson University >>>>>>>>>>>>>> Clemson, SC >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> -- >>>>>>>>>>>>>> Becky Ligon >>>>>>>>>>>>>> Research Associate >>>>>>>>>>>>>> Clemson University >>>>>>>>>>>>>> Clemson, SC >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> -- >>>>>>>>>>>>>> Becky Ligon >>>>>>>>>>>>>> Research Associate >>>>>>>>>>>>>> Clemson University >>>>>>>>>>>>>> Clemson, SC >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> -- >>>>>>>>>>>>> Becky Ligon >>>>>>>>>>>>> Research Associate >>>>>>>>>>>>> Clemson University >>>>>>>>>>>>> Clemson, SC >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> -- >>>>>>>>>>>>> Becky Ligon >>>>>>>>>>>>> Research Associate >>>>>>>>>>>>> Clemson University >>>>>>>>>>>>> Clemson, SC >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>>> Becky Ligon >>>>>>>>>>>> Research Associate >>>>>>>>>>>> Clemson University >>>>>>>>>>>> Clemson, SC >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>>> Becky Ligon >>>>>>>>>>>> Research Associate >>>>>>>>>>>> Clemson University >>>>>>>>>>>> Clemson, SC >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> Becky Ligon >>>>>>>>>>> Research Associate >>>>>>>>>>> Clemson University >>>>>>>>>>> Clemson, SC >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> Becky Ligon >>>>>>>>>>> Research Associate >>>>>>>>>>> Clemson University >>>>>>>>>>> Clemson, SC >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> Becky Ligon >>>>>>>>>>> Research Associate >>>>>>>>>>> Clemson University >>>>>>>>>>> Clemson, SC >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> Becky Ligon >>>>>>>>>> Research Associate >>>>>>>>>> Clemson University >>>>>>>>>> Clemson, SC >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Becky Ligon >>>>>>>>> Research Associate >>>>>>>>> Clemson University >>>>>>>>> Clemson, SC >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Becky Ligon >>>>>>>> Research Associate >>>>>>>> Clemson University >>>>>>>> Clemson, SC >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Becky Ligon >>>>>>> Research Associate >>>>>>> Clemson University >>>>>>> Clemson, SC >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Becky Ligon >>>>>> Research Associate >>>>>> Clemson University >>>>>> Clemson, SC >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Becky Ligon >>>>>> Research Associate >>>>>> Clemson University >>>>>> Clemson, SC >>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> Becky Ligon >>>>> Research Associate >>>>> Clemson University >>>>> Clemson, SC >>>>> >>>>> >>>>> >>>>> -- >>>>> Becky Ligon >>>>> Research Associate >>>>> Clemson University >>>>> Clemson, SC >>>> >>>> >>>> >>>> >>>> -- >>>> Becky Ligon >>>> Research Associate >>>> Clemson University >>>> Clemson, SC >>>> >>>> >>>> >>>> -- >>>> Becky Ligon >>>> Research Associate >>>> Clemson University >>>> Clemson, SC >>> >>> >>> >>> >>> -- >>> Becky Ligon >>> Research Associate >>> Clemson University >>> Clemson, SC >> _______________________________________________ Pvfs2-users mailing list [email protected] http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users
