Hi all, I'll probably be told to take this up in the user's list, but I've had bad experiences with 'user' lists. I figured this time, I'd go straight to the guys in-the-know.
I wasn't quite sure how to title this post given the nature of my problem. My cluster is made up of some machines which are i386-based, running 2.6.13-1.1532_FC4 and 2.6.14-1.1653_FC4smp, and some machines which are x86_64-based, running 2.6.11.11smp. I/O and Metaservers are running 2.6.13-1.1532_FC4. Here is what I have found: Given this example file /scratch/xxx/file where /scratch is our PVFS2-1.3.0 volume and an identical file stored on ext3 /home/student/x/xxx/file, I get different results from the stat command: [EMAIL PROTECTED] ~]$ stat /scratch/xxx/file File: `/scratch/xxx/file' Size: 150263 Blocks: 296 IO Block: 4194304 regular file Device: 13h/19d Inode: 1046003 Links: 1 Access: (0644/-rw-r--r--) Uid: (1288406/xxx) Gid: ( 100/ users) Access: 2006-01-10 16:48:35.000000000 -0500 Modify: 2006-01-10 16:48:35.000000000 -0500 Change: 2006-01-10 16:48:35.000000000 -0500 [EMAIL PROTECTED] ~]$ stat /home/student/x/xxx/file File: `/home/student/x/xxx/file' Size: 150263 Blocks: 304 IO Block: 4096 regular file Device: 811h/2065d Inode: 63047415 Links: 1 Access: (0644/-rw-r--r--) Uid: (1229806/xxx) Gid: ( 100/ users) Access: 2006-01-19 21:24:41.000000000 -0500 Modify: 2006-01-19 21:18:02.000000000 -0500 Change: 2006-01-19 21:18:02.000000000 -0500 even though 'diff' shows no differences between the two files. >From this, I get: /scratch/xxx/file: Block size = 296 - File size = 147456 /home/student/x/xxx/file: Block size = 304 - File size = 150263 Blocks (stat) Sizes (ls -l) 296/304 ~= 147456/150263 where (296/304 = 0.973584) and (147456/150263 = 0.981319) showing that the ratios are roughly equivalent which lead me to investigate the stat, cp, and cat commands. Copying the file from PVFS2 causes the file to be truncated upon being written to disk. The 'cp' command must call xstat() or lxstat() to get the number of blocks from a particular file, assumedly using that information to know "how much" to copy. On the otherhand cat /scratch/xxx/file > /home/student/x/xxx/file seems to work perfectly, calling fxstat(). Running 'strings' on 'cp' and 'cat' seems to point out a difference in the system calls as 'cat' has a single reference to fxstat() while 'cp' references xstat(), fxstat(), lxstat(). The stat command only references lxstat() and xstat() so I assume that fxstat() must be "safe" for this file at least since 'cat' and 'diff' work, but 'cp' and 'stat' are consistently wrong with respect to this particular file. Now, the point of all this is that I am curious as to why PVFS2 is recording/reporting false Block sizes in/from the 'inode table' (not sure if PVFS2 even has an inode table or if its emulated). Is this a bug, an error on my part or has this already been fixed and I probably should have asked for a ChangeLog instead? In summary: 1) Some files copy cleanly from PVFS2 to other file systems while others do not. 2) The difference in block size reported by stat is proportional to the difference in file size reported by 'ls' even though 'cat'ing the file or viewing the file while ON PVFS2 shows that all the contents are accounted for. 3) Using the 'cat' command seems to avoid the problem (possibly since it avoids making a call to lxstat() or xstat()) but I don't want to push this as a fix for our users (nor would any of you, I'd assume). 4) My PVFS2 I/O and Metaservers are running on AthlonXP's while the client in question is running on Opterons. 5) Files "appear" different using stat when in fact they are the same. Copying with 'cp' yields truncated files being written. The difference in size of the original file from the copy is proportional to the discrepancies observed with 'stat'. Thanks for any help. -Brian Smith _______________________________________________ Pvfs2-developers mailing list [email protected] http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers
