Dear Phil,
I can't test the setfattr command on a local filesystem as far as I can
tell:
$ cd /var/tmp
$ mkdir test
$ setfattr -n "user.pvfs2.dist_name" -v "varstrip_dist" test
setfattr: test: Operation not supported
$
...or is there a test I can do on an ext3 filesystem dir?
I changed the value in include/pvfs2-types.h to 16K (should be plenty
big enough I guess...) & recompiled - now setfattr works the first time,
but fails subsequently..
Note: testing on a single RHEL5 node, if that makes a difference
$ mkdir test
$ getfattr test
$ setfattr -n "user.pvfs2.dist_name" -v "varstrip_dist" test
$ getfattr test
# file: test
user.pvfs2.dist_name
$ getfattr -n "user.pvfs2.dist_name" test
# file: test
user.pvfs2.dist_name="varstrip_dist"
$ setfattr -n "user.pvfs2.dist_name" -v "varstrip_dist" test
$ getfattr -n "user.pvfs2.dist_name" test
# file: test
user.pvfs2.dist_name="varstrip_dist"
$
$ setfattr -n "user.pvfs2.dist_params" -v
"strips:0:16K;1:16K;2:16K;3:16K;4:16K;5:16K;6:16K;7:16K;8:16K;9:16K;10:16K;11:16K;12:16K;13:16K;14:16K;15:16K;16:16K;17:16K;18:16K;19:16K;20:16K;21:16K;22:16K;23:16K;24:16K;25:16K;26:16K;27:16K;28:16K;29:16K;30:16K;31:16K;32:16K;33:16K;34:16K;35:16K;36:16K;37:16K;38:16K;39:16K;40:16K;41:16K;42:16K;43:16K;44:16K;45:16K;46:16K;47:16K;48:16K;49:16K;50:16K;51:16K;52:16K;53:16K;54:16K;55:16K;56:16K;57:16K;58:16K;59:16K;60:16K;61:16K;62:16K;63:16K"
test
$ getfattr test
# file: test
user.pvfs2.dist_name
user.pvfs2.dist_params
$ getfattr -n "user.pvfs2.dist_params" test
# file: test
user.pvfs2.dist_params="strips:0:16K;1:16K;2:16K;3:16K;4:16K;5:16K;6:16K;7:16K;8:16K;9:16K;10:16K;11:16K;12:16K;13:16K;14:16K;15:16K;16:16K;17:16K;18:16K;19:16K;20:16K;21:16K;22:16K;23:16K;24:16K;25:16K;26:16K;27:16K;28:16K;29:16K;30:16K;31:16K;32:16K;33:16K;34:16K;35:16K;36:16K;37:16K;38:16K;39:16K;40:16K;41:16K;42:16K;43:16K;44:16K;45:16K;46:16K;47:16K;48:16K;49:16K;50:16K;51:16K;52:16K;53:16K;54:16K;55:16K;56:16K;57:16K;58:16K;59:16K;60:16K;61:16K;62:16K;63:16K"
$ setfattr -n "user.pvfs2.dist_params" -v
"strips:0:16K;1:16K;2:16K;3:16K;4:16K;5:16K;6:16K;7:16K;8:16K;9:16K;10:16K;11:16K;12:16K;13:16K;14:16K;15:16K;16:16K;17:16K;18:16K;19:16K;20:16K;21:16K;22:16K;23:16K;24:16K;25:16K;26:16K;27:16K;28:16K;29:16K;30:16K;31:16K;32:16K;33:16K;34:16K;35:16K;36:16K;37:16K;38:16K;39:16K;40:16K;41:16K;42:16K;43:16K;44:16K;45:16K;46:16K;47:16K;48:16K;49:16K;50:16K;51:16K;52:16K;53:16K;54:16K;55:16K;56:16K;57:16K;58:16K;59:16K;60:16K;61:16K;62:16K;63:16K"
test
setfattr: test: Invalid argument
$ setfattr -n "user.pvfs2.dist_params" -v "strips:0:16K" test
setfattr: test: Invalid argument
$
I cannot write anything in the directory (though note, the configuration is
invalid for a single node, so that may be reasonable)
$ cd test/
$ dd if=/dev/urandom of=testfile bs=16k count=16
dd: opening `testfile': Invalid argument
$ cd ..
At this point the only way to fix the settings seems to be to remove
and recreate the directory
"dd" does work with a valid single node configuration, however:
$ rm -rf test/
$ mkdir test
$ setfattr -n "user.pvfs2.dist_name" -v "varstrip_dist" test
$ setfattr -n "user.pvfs2.dist_params" -v "strips:0:16K" test
$ getfattr test
# file: test
user.pvfs2.dist_name
user.pvfs2.dist_params
$ getfattr -n "user.pvfs2.dist_name" test
# file: test
user.pvfs2.dist_name="varstrip_dist"
$ getfattr -n "user.pvfs2.dist_params" test
# file: test
user.pvfs2.dist_params="strips:0:16K"
$ setfattr -n "user.pvfs2.dist_params" -v "strips:0:16K" test
$ getfattr -n "user.pvfs2.dist_params" test
# file: test
user.pvfs2.dist_params="strips:0:16K"
$ cd test
$ dd if=/dev/urandom of=testfile bs=16k count=16
16+0 records in
16+0 records out
262144 bytes (262 kB) copied, 0.162831 seconds, 1.6 MB/s
$ ls -l testfile
-rw-r--r-- 1 tonykew ccrstaff 262144 Oct 9 14:47 testfile
$ cd ..
$
It is not easy for me to test this code on 64 nodes.
I don't know if we can go further with this without me doing so?
Tony
Tony Kew
SAN Administrator
The Center for Computational Research
New York State Center of Excellence
in Bioinformatics & Life Sciences
701 Ellicott Street, Buffalo, NY 14203
CoE Office: (716) 881-8930 Fax: (716) 849-6656
CSE Office: (716) 645-3797 x2174
Cell: (716) 560-0910 Home: (716) 874-2126
"I love deadlines, I love the whooshing noise they make as they go by."
Douglas Adams
Phil Carns wrote:
Hi Tony,
Looking at the length of those strings, it looks like the problem
starts when the xattr value is 256 characters. Digging around in the
code it looks like we have a hard coded limit on this in PVFS (from
pvfs2-types.h):
#define PVFS_MAX_XATTR_VALUELEN 256
I don't know if there is a good reason for this limit or if it is
arbitrary. Could you test and see if you can set an extended
attribute with that value on a local file system on your box?
I don't think we have had anyone try to specify varstrip parameters
for that many datafiles before :)
thanks,
-Phil
Tony Kew wrote:
Dear All,
I'm running pvfs 7.2.1 with Murali Vilayannur xarrts patch from this
message thread:
http://osdir.com/ml/file-systems.pvfs2.users/2007-01/msg00050.html
(which, if memory serves, Phil Carns put into CVS)
I have a PVFS filesystem over 64 nodes, mounted on all those nodes:
$ pvfs2-ping -m /scratch/1076995.bono.ccr.buffalo.edu/mnt
[...]
(6) Verifying that fsid 1076995 is acceptable to all servers...
Ok; all servers understand fs_id 1076995
(7) Verifying that root handle is owned by one server...
Root handle: 1048576
Ok; root handle is owned by exactly one server.
=============================================================
The PVFS2 filesystem at /scratch/1076995.bono.ccr.buffalo.edu/mnt
appears to be correctly configured.
$
All nodes 64 nodes are up:
$ pvfs2-ping -m /scratch/1076995.bono.ccr.buffalo.edu/mnt | sed -n
'/^(5)/,/^(6)/p' | sed -n '/data servers/,/^$/p' | sed 1d | sed '$d'
| wc -l
64
$
I'm trying to create a varstrip_dist dir with (effectively) a stripe
across all 64 nodes,
with 16K on each node, but the setfattr fails for 37 (or more) nodes:
$ mount | grep pvfs2
tcp://c27n20:3334/pvfs2-fs-1076995.bono.ccr.buffalo.edu on
/scratch/1076995.bono.ccr.buffalo.edu/mnt type pvfs2 (rw)
$ cd /scratch/1076995.bono.ccr.buffalo.edu/mnt
$ mkdir test
$ setfattr -n "user.pvfs2.dist_name" -v "varstrip_dist" test
$ getfattr test
# file: test
user.pvfs2.dist_name
$ getfattr -n "user.pvfs2.dist_name" test
# file: test
user.pvfs2.dist_name="varstrip_dist"
$ setfattr -n "user.pvfs2.dist_params" -v
"strips:0:16K;1:16K;2:16K;3:16K;4:16K;5:16K;6:16K;7:16K;8:16K;9:16K;10:16K;11:16K;12:16K;13:16K;14:16K;15:16K;16:16K;17:16K;18:16K;19:16K;20:16K;21:16K;22:16K;23:16K;24:16K;25:16K;26:16K;27:16K;28:16K;29:16K;30:16K;31:16K;32:16K;33:16K;34:16K;35:16K;36:16K"
test
$ getfattr -n "user.pvfs2.dist_params" test
# file: test
user.pvfs2.dist_params="strips:0:16K;1:16K;2:16K;3:16K;4:16K;5:16K;6:16K;7:16K;8:16K;9:16K;10:16K;11:16K;12:16K;13:16K;14:16K;15:16K;16:16K;17:16K;18:16K;19:16K;20:16K;21:16K;22:16K;23:16K;24:16K;25:16K;26:16K;27:16K;28:16K;29:16K;30:16K;31:16K;32:16K;33:16K;34:16K;35:16K;36:16K"
$ setfattr -n "user.pvfs2.dist_params" -v
"strips:0:16K;1:16K;2:16K;3:16K;4:16K;5:16K;6:16K;7:16K;8:16K;9:16K;10:16K;11:16K;12:16K;13:16K;14:16K;15:16K;16:16K;17:16K;18:16K;19:16K;20:16K;21:16K;22:16K;23:16K;24:16K;25:16K;26:16K;27:16K;28:16K;29:16K;30:16K;31:16K;32:16K;33:16K;34:16K;35:16K;36:16K;37:16K"
test
setfattr: test: Invalid argument
$ setfattr -n "user.pvfs2.dist_params" -v
"strips:0:16K;1:16K;2:16K;3:16K;4:16K;5:16K;6:16K;7:16K;8:16K;9:16K;10:16K;11:16K;12:16K;13:16K;14:16K;15:16K;16:16K;17:16K;18:16K;19:16K;20:16K;21:16K;22:16K;23:16K;24:16K;25:16K;26:16K;27:16K;28:16K;29:16K;30:16K;31:16K;32:16K;33:16K;34:16K;35:16K;36:16K;37:16K;38:16K;39:16K;40:16K;41:16K;42:16K;43:16K;44:16K;45:16K;46:16K;47:16K;48:16K;49:16K;50:16K;51:16K;52:16K;53:16K;54:16K;55:16K;56:16K;57:16K;58:16K;59:16K;60:16K;61:16K;62:16K;63:16K"
test
setfattr: test: Invalid argument
$
Sorry for the over long lines (I made sure its not a problem with
line wrapping).
Since the error is from setfattr I wonder if this is a RedHat problem...
I'm running RedHat Enterprise Linux AS v.4 update 5 on these nodes, and
the attr package (which includes setfattr) is the latest version from
RH:
$ rpm -qf `type -p setfattr`
attr-2.4.16-3.1.el4
$
Thanks Much,
Tony
Tony Kew
SAN Administrator
The Center for Computational Research
New York State Center of Excellence
in Bioinformatics & Life Sciences
701 Ellicott Street, Buffalo, NY 14203
CoE Office: (716) 881-8930 Fax: (716) 849-6656
CSE Office: (716) 645-3797 x2174
Cell: (716) 560-0910 Home: (716) 874-2126
"I love deadlines, I love the whooshing noise they make as they go by."
Douglas Adams
_______________________________________________
Pvfs2-users mailing list
[email protected]
http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users
_______________________________________________
Pvfs2-users mailing list
[email protected]
http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users
