Peter Cordes wrote: >On Wed, Aug 27, 2003 at 04:26:56AM +0900, Kyungwon Chun wrote: > > >>I made my new cluster using Sarge. The problem is that I can not treat >>a file bigger than 2 GB. I'm trying to make a file on NFS mounted >>directory, using mpich and hdf5. The error message is >> >>p15_4159: p4_error: : 1 >>File locking failed in ADIOI_Set_lock. If the file system is NFS, you >>need to use NFS version 3 and mount the directory with the 'noac' option >>(no attribute caching). >> >>But, I'm mounted the directory using version 3 NFS with no attribute >>cache. (I found that the locking function is not work properly with the >>nfs-common package of Sarge. So, I used that one from the Sid by >>compiling from the source package.) I also tried the suggestion of HDF5 >>installation manual i.e. >>adding the following compiler option when building the mpich package. >> >>-cflags="-D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64" >> >> >>But, It seems that still not working. Is there any suggestion? >> >> > > Can you try on an ext3 (or anything that isn't NFS) to see if the problem >is due to NFS? I just created a > 2GB file on ext3, over NFS, using >dd if=/dev/zero of=bigfile bs=1024k count=2200 >I'm appending to it with cat, and it's now up to 3.4GB. Big files don't >seem to be a problem for NFS on Linux. I'm using Linux 2.4.22 on the client >and server, with the NFS kernel server. Maybe your problem is that lockd >isn't running on the server, or something like that? Anyway, I don't think >the problem is just because of large files. > > > I did the same test on my NFS mounted directory (dd if=/dev/zero of=bigfile bs=1024k count=2200). I could make 2.2GB file with this method without any problem. I also think that big files don't seem to be a problem for NFS on Linux. But, If I try to make >2GB file with MPICH, It make a problem. I also, check that the lockd daemon in running on the server. The other possible cause of the problem is HDF5 library. But, HDF5 library also works fine on a host filesystem. I checked this using the test programs in the source package and my own one. Now, I think the problem originated from MPICH package.
The information of my system follows : SERVER $ uname -a Linux magi 2.4.21-4-686 #1 Sat Aug 2 23:27:25 EST 2003 i686 GNU/Linux $ rpcinfo -p program vers proto port 100000 2 tcp 111 portmapper 100000 2 udp 111 portmapper 100004 2 udp 978 ypserv 100004 1 udp 978 ypserv 100004 2 tcp 981 ypserv 100004 1 tcp 981 ypserv 100009 1 udp 980 yppasswdd 600100069 1 udp 983 600100069 1 tcp 985 100007 2 udp 990 ypbind 100007 1 udp 990 ypbind 100007 2 tcp 993 ypbind 100007 1 tcp 993 ypbind 391002 1 tcp 32768 sgi_fam 391002 2 tcp 32768 sgi_fam 100003 2 udp 2049 nfs 100003 3 udp 2049 nfs 100003 2 tcp 2049 nfs 100003 3 tcp 2049 nfs 100021 1 udp 32950 nlockmgr 100021 3 udp 32950 nlockmgr 100021 4 udp 32950 nlockmgr 100021 1 tcp 38192 nlockmgr 100021 3 tcp 38192 nlockmgr 100021 4 tcp 38192 nlockmgr 100005 1 udp 893 mountd 100005 1 tcp 896 mountd 100005 2 udp 893 mountd 100005 2 tcp 896 mountd 100005 3 udp 893 mountd 100005 3 tcp 896 mountd 100024 1 udp 913 status 100024 1 tcp 916 status $ /usr/sbin/nfsstat Server rpc stats: calls badcalls badauth badclnt xdrcall 10428820 0 0 0 0 Server nfs v2: null getattr setattr root lookup readlink 0 0% 2943386 74% 0 0% 0 0% 779826 19% 6113 0% read wrcache write create remove rename 163391 4% 0 0% 0 0% 0 0% 0 0% 0 0% link symlink mkdir rmdir readdir fsstat 0 0% 0 0% 0 0% 0 0% 40481 1% 66 0% Server nfs v3: null getattr setattr lookup access readlink 20 0% 1536536 23% 43233 0% 128242 1% 1017021 15% 5864 0% read write create mkdir symlink mknod 108858 1% 3469747 53% 33793 0% 105 0% 0 0% 0 0% remove rmdir rename link readdir readdirplus 32027 0% 43 0% 1003 0% 133 0% 2277 0% 0 0% fsstat fsinfo pathconf commit 839 0% 839 0% 0 0% 114977 1% CLIENT $ uname -a Linux eva1 2.4.21 #1 Sun Aug 24 15:43:58 KST 2003 i686 GNU/Linux $ rpcinfo -p program vers proto port 100000 2 tcp 111 portmapper 100000 2 udp 111 portmapper 100021 1 udp 32768 nlockmgr 100021 3 udp 32768 nlockmgr 100021 4 udp 32768 nlockmgr 100007 2 udp 813 ypbind 100007 1 udp 813 ypbind 100007 2 tcp 816 ypbind 100007 1 tcp 816 ypbind 100024 1 udp 842 status 100024 1 tcp 845 status $ /usr/sbin/nfsstat Warning: /proc/net/rpc/nfsd: No such file or directory Server rpc stats: calls badcalls badauth badclnt xdrcall 0 0 0 0 0 Server nfs v2: null getattr setattr root lookup readlink 0 0% 0 0% 0 0% 0 0% 0 0% 0 0% read wrcache write create remove rename 0 0% 0 0% 0 0% 0 0% 0 0% 0 0% link symlink mkdir rmdir readdir fsstat 0 0% 0 0% 0 0% 0 0% 0 0% 0 0% Client rpc stats: calls retrans authrefrsh 1607040 10298 0 Client nfs v2: null getattr setattr root lookup readlink 0 0% 166851 77% 0 0% 0 0% 41090 19% 234 0% read wrcache write create remove rename 5564 2% 0 0% 0 0% 0 0% 0 0% 0 0% link symlink mkdir rmdir readdir fsstat 0 0% 0 0% 0 0% 0 0% 1631 0% 1 0% Client nfs v3: null getattr setattr lookup access readlink 0 0% 64453 4% 2169 0% 4554 0% 53622 3% 79 0% read write create mkdir symlink mknod 2270 0% 1257097 90% 1516 0% 2 0% 0 0% 0 0% remove rmdir rename link readdir readdirplus 1443 0% 0 0% 47 0% 6 0% 61 0% 0 0% fsstat fsinfo pathconf commit 90 0% 90 0% 0 0% 4170 0% $ mount rootfs on / type rootfs (rw) /dev/root on / type nfs (ro,v2,rsize=4096,wsize=4096,hard,udp,nolock,addr=192.168.0.1) none on /dev type devfs (rw) none on /proc type proc (rw) 192.168.0.1://var/lib/diskless/default/192.168.0.11/etc on /etc type nfs (rw,v3,rsize=8192,wsize=8192,hard,udp,nolock,addr=192.168.0.1) 192.168.0.1:/var/lib/diskless/default/192.168.0.11/etc on /etc type nfs (rw,v3,rsize=8192,wsize=8192,hard,udp,nolock,addr=192.168.0.1) 192.168.0.1:/var/lib/diskless/default/192.168.0.11/var on /var type nfs (rw,v3,rsize=8192,wsize=8192,hard,udp,nolock,addr=192.168.0.1) 192.168.0.1:/var/lib/diskless/default/192.168.0.11/tmp on /tmp type nfs (rw,v3,rsize=8192,wsize=8192,hard,udp,nolock,addr=192.168.0.1) 192.168.0.1:/var/lib/diskless/default/root/var/lib/dpkg on /var/lib/dpkg type nfs (ro,v3,rsize=8192,wsize=8192,hard,udp,nolock,addr=192.168.0.1) 192.168.0.1:/home on /home type nfs (rw,sync,v3,rsize=8192,wsize=8192,acregmin=0,acregmax=0,acdirmin=0,acdirmax=0,hard,udp,noac,lock,addr=192.168.0.1) -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
