On Jan 19, 2006, at 10:41 PM, Brian R. Smith wrote:
Hi all,
I'll probably be told to take this up in the user's list, but I've
had bad
experiences with 'user' lists. I figured this time, I'd go
straight to the
guys in-the-know.
Hi Brian,
I think you may find the pvfs2-users list responsive and the
subscribers fairly knowledgeable. There's also more subscribers
there than here I believe, so you might have a higher chance of a
quicker response.
I wasn't quite sure how to title this post given the nature of my
problem. My
cluster is made up of some machines which are i386-based, running
2.6.13-1.1532_FC4 and 2.6.14-1.1653_FC4smp, and some machines which
are
x86_64-based, running 2.6.11.11smp. I/O and Metaservers are running
2.6.13-1.1532_FC4.
Here is what I have found:
Given this example file /scratch/xxx/file where /scratch is our
PVFS2-1.3.0
volume and an identical file stored on ext3 /home/student/x/xxx/
file, I get
different results from the stat command:
IIRC we fixed a truncate bug in the 1.3.1 release. (Note that the
latest is 1.3.2). If possible please update to the latest release.
[EMAIL PROTECTED] ~]$ stat /scratch/xxx/file
File: `/scratch/xxx/file'
Size: 150263 Blocks: 296 IO Block: 4194304
regular file
Device: 13h/19d Inode: 1046003 Links: 1
Access: (0644/-rw-r--r--) Uid: (1288406/xxx) Gid: ( 100/ users)
Access: 2006-01-10 16:48:35.000000000 -0500
Modify: 2006-01-10 16:48:35.000000000 -0500
Change: 2006-01-10 16:48:35.000000000 -0500
[EMAIL PROTECTED] ~]$ stat /home/student/x/xxx/file
File: `/home/student/x/xxx/file'
Size: 150263 Blocks: 304 IO Block: 4096 regular
file
Device: 811h/2065d Inode: 63047415 Links: 1
Access: (0644/-rw-r--r--) Uid: (1229806/xxx) Gid: ( 100/ users)
Access: 2006-01-19 21:24:41.000000000 -0500
Modify: 2006-01-19 21:18:02.000000000 -0500
Change: 2006-01-19 21:18:02.000000000 -0500
even though 'diff' shows no differences between the two files.
From this, I get:
/scratch/xxx/file: Block size = 296 - File size = 147456
/home/student/x/xxx/file: Block size = 304 - File size = 150263
Blocks (stat) Sizes (ls -l)
296/304 ~= 147456/150263
where (296/304 = 0.973584) and (147456/150263 = 0.981319) showing
that the
ratios are roughly equivalent which lead me to investigate the
stat, cp, and
cat commands. Copying the file from PVFS2 causes the file to be
truncated
upon being written to disk.
This looks very similar to a bug we had in 1.3.0 (fixed in 1.3.1 and
1.3.2).
You will see some minor differences in the results from fstat on a
pvfs2 mount than elsewhere. Murali was explaining to me the reason
for this the other day, but after the weekend I don't remember the
details :-). I'll leave it to him to elaborate.
-sam
The 'cp' command must call xstat() or lxstat() to get the number of
blocks
from a particular file, assumedly using that information to know
"how much"
to copy. On the otherhand
cat /scratch/xxx/file > /home/student/x/xxx/file
seems to work perfectly, calling fxstat(). Running 'strings' on
'cp' and
'cat' seems to point out a difference in the system calls as 'cat'
has a
single reference to fxstat() while 'cp' references xstat(), fxstat(),
lxstat(). The stat command only references lxstat() and xstat() so
I assume
that fxstat() must be "safe" for this file at least since 'cat' and
'diff'
work, but 'cp' and 'stat' are consistently wrong with respect to this
particular file.
Now, the point of all this is that I am curious as to why PVFS2 is
recording/reporting false Block sizes in/from the 'inode
table' (not sure if
PVFS2 even has an inode table or if its emulated). Is this a bug,
an error
on my part or has this already been fixed and I probably should
have asked
for a ChangeLog instead?
In summary:
1) Some files copy cleanly from PVFS2 to other file systems while
others do
not.
2) The difference in block size reported by stat is proportional to
the
difference in file size reported by 'ls' even though 'cat'ing the
file or
viewing the file while ON PVFS2 shows that all the contents are
accounted
for.
3) Using the 'cat' command seems to avoid the problem (possibly
since it
avoids making a call to lxstat() or xstat()) but I don't want to
push this as
a fix for our users (nor would any of you, I'd assume).
4) My PVFS2 I/O and Metaservers are running on AthlonXP's while the
client in
question is running on Opterons.
5) Files "appear" different using stat when in fact they are the same.
Copying with 'cp' yields truncated files being written. The
difference in
size of the original file from the copy is proportional to the
discrepancies
observed with 'stat'.
Thanks for any help.
-Brian Smith
_______________________________________________
Pvfs2-developers mailing list
[email protected]
http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers
_______________________________________________
Pvfs2-developers mailing list
[email protected]
http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers
