Hello again, Just to let you know, we've observed this same behavior on another cluster, with uniform 32-bit Intel Xeon processors, all running 2.6.14-1.1653_FC4smp.
Again, thank you again for an help and please let me know if you want any more information. -Brian On Thursday 19 January 2006 23:41, Brian R. Smith wrote: > Hi all, > > I'll probably be told to take this up in the user's list, but I've had bad > experiences with 'user' lists. I figured this time, I'd go straight to the > guys in-the-know. > > I wasn't quite sure how to title this post given the nature of my problem. > My cluster is made up of some machines which are i386-based, running > 2.6.13-1.1532_FC4 and 2.6.14-1.1653_FC4smp, and some machines which are > x86_64-based, running 2.6.11.11smp. I/O and Metaservers are running > 2.6.13-1.1532_FC4. > > Here is what I have found: > > Given this example file /scratch/xxx/file where /scratch is our PVFS2-1.3.0 > volume and an identical file stored on ext3 /home/student/x/xxx/file, I get > different results from the stat command: > > [EMAIL PROTECTED] ~]$ stat /scratch/xxx/file > File: `/scratch/xxx/file' > Size: 150263 Blocks: 296 IO Block: 4194304 regular file > Device: 13h/19d Inode: 1046003 Links: 1 > Access: (0644/-rw-r--r--) Uid: (1288406/xxx) Gid: ( 100/ users) > Access: 2006-01-10 16:48:35.000000000 -0500 > Modify: 2006-01-10 16:48:35.000000000 -0500 > Change: 2006-01-10 16:48:35.000000000 -0500 > > [EMAIL PROTECTED] ~]$ stat /home/student/x/xxx/file > File: `/home/student/x/xxx/file' > Size: 150263 Blocks: 304 IO Block: 4096 regular file > Device: 811h/2065d Inode: 63047415 Links: 1 > Access: (0644/-rw-r--r--) Uid: (1229806/xxx) Gid: ( 100/ users) > Access: 2006-01-19 21:24:41.000000000 -0500 > Modify: 2006-01-19 21:18:02.000000000 -0500 > Change: 2006-01-19 21:18:02.000000000 -0500 > > even though 'diff' shows no differences between the two files. > > >From this, I get: > > /scratch/xxx/file: Block size = 296 - File size = 147456 > /home/student/x/xxx/file: Block size = 304 - File size = 150263 > > Blocks (stat) Sizes (ls -l) > 296/304 ~= 147456/150263 > > where (296/304 = 0.973584) and (147456/150263 = 0.981319) showing that the > ratios are roughly equivalent which lead me to investigate the stat, cp, > and cat commands. Copying the file from PVFS2 causes the file to be > truncated upon being written to disk. > > The 'cp' command must call xstat() or lxstat() to get the number of blocks > from a particular file, assumedly using that information to know "how much" > to copy. On the otherhand > > cat /scratch/xxx/file > /home/student/x/xxx/file > > seems to work perfectly, calling fxstat(). Running 'strings' on 'cp' and > 'cat' seems to point out a difference in the system calls as 'cat' has a > single reference to fxstat() while 'cp' references xstat(), fxstat(), > lxstat(). The stat command only references lxstat() and xstat() so I > assume that fxstat() must be "safe" for this file at least since 'cat' and > 'diff' work, but 'cp' and 'stat' are consistently wrong with respect to > this particular file. > > Now, the point of all this is that I am curious as to why PVFS2 is > recording/reporting false Block sizes in/from the 'inode table' (not sure > if PVFS2 even has an inode table or if its emulated). Is this a bug, an > error on my part or has this already been fixed and I probably should have > asked for a ChangeLog instead? > > In summary: > > 1) Some files copy cleanly from PVFS2 to other file systems while others do > not. > > 2) The difference in block size reported by stat is proportional to the > difference in file size reported by 'ls' even though 'cat'ing the file or > viewing the file while ON PVFS2 shows that all the contents are accounted > for. > > 3) Using the 'cat' command seems to avoid the problem (possibly since it > avoids making a call to lxstat() or xstat()) but I don't want to push this > as a fix for our users (nor would any of you, I'd assume). > > 4) My PVFS2 I/O and Metaservers are running on AthlonXP's while the client > in question is running on Opterons. > > 5) Files "appear" different using stat when in fact they are the same. > Copying with 'cp' yields truncated files being written. The difference in > size of the original file from the copy is proportional to the > discrepancies observed with 'stat'. > > Thanks for any help. > > > -Brian Smith > > > > > _______________________________________________ > Pvfs2-developers mailing list > [email protected] > http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers -- Brian R. Smith HPC Systems Administrator, Research Computing University of South Florida 4202 E Fowler Ave. LIB 608 Tampa, FL. 33620 1(813)974-1467 http://rc.usf.edu _______________________________________________ Pvfs2-developers mailing list [email protected] http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers
