Dear Phil,
The patch looks good - I can set a 64 node config now:
e.g.
ramones$ setfattr -n "user.pvfs2.dist_name" -v "varstrip_dist" test
ramones$ setfattr -n "user.pvfs2.dist_params" -v
"strips:0:16K;1:16K;2:16K;3:16K;4:16K;5:16K;6:16K;7:16K;8:16K;9:16K;10:16K;11:16K;12:16K;13:16K;14:16K;15:16K;16:16K;17:16K;18:16K;19:16K;20:16K;21:16K;22:16K;23:16K;24:16K;25:16K;26:16K;27:16K;28:16K;29:16K;30:16K;31:16K;32:16K;33:16K;34:16K;35:16K;36:16K;37:16K;38:16K;39:16K;40:16K;41:16K;42:16K;43:16K;44:16K;45:16K;46:16K;47:16K;48:16K;49:16K;50:16K;51:16K;52:16K;53:16K;54:16K;55:16K;56:16K;57:16K;58:16K;59:16K;60:16K;61:16K;62:16K;63:16K"
test
ramones$ getfattr -n "user.pvfs2.dist_params" test
# file: test
user.pvfs2.dist_params="strips:0:16K;1:16K;2:16K;3:16K;4:16K;5:16K;6:16K;7:16K;8:16K;9:16K;10:16K;11:16K;12:16K;13:16K;14:16K;15:16K;16:16K;17:16K;18:16K;19:16K;20:16K;21:16K;22:16K;23:16K;24:16K;25:16K;26:16K;27:16K;28:16K;29:16K;30:16K;31:16K;32:16K;33:16K;34:16K;35:16K;36:16K;37:16K;38:16K;39:16K;40:16K;41:16K;42:16K;43:16K;44:16K;45:16K;46:16K;47:16K;48:16K;49:16K;50:16K;51:16K;52:16K;53:16K;54:16K;55:16K;56:16K;57:16K;58:16K;59:16K;60:16K;61:16K;62:16K;63:16K"
ramones$
It may take a little while till I can install this on the cluster & test
PVFSv2
over 64 nodes, but at least the parameter can be set :-)
Thanks,
Tony
Tony Kew
SAN Administrator
The Center for Computational Research
New York State Center of Excellence
in Bioinformatics & Life Sciences
701 Ellicott Street, Buffalo, NY 14203
CoE Office: (716) 881-8930 Fax: (716) 849-6656
CSE Office: (716) 645-3797 x2174
Cell: (716) 560-0910 Home: (716) 874-2126
"I love deadlines, I love the whooshing noise they make as they go by."
Douglas Adams
Phil Carns wrote:
Hi Tony,
Would you mind trying the attached patch? It increases the maximum
xattr value size to 8KB and also fixes a problem further down in the
code regarding how to handle xattrs that large. The latter part of
the patch should solve the problem that you ran into in which you
could only set a large xattr once.
thanks,
-Phil
Tony Kew wrote:
Dear Phil,
I verified that this is not an issue with the underlying filesystem:
c23n12$ mount | grep \ /scratch\
/dev/sda3 on /scratch type ext2 (rw)
c23n12$ sudo umount /scratch
c23n12$ sudo mount -o rw,user_xattr /dev/sda3 /scratch
c23n12$ mount | grep \ /scratch\
/dev/sda3 on /scratch type ext2 (rw,user_xattr)
c23n12$ cd /scratch
c23n12$ mkdir test
c23n12$ setfattr -n "user.pvfs2.dist_name" -v "varstrip_dist" test
c23n12$ getfattr test/
# file: test
user.pvfs2.dist_name
c23n12$ setfattr -n "user.pvfs2.dist_params" -v
"strips:0:16K;1:16K;2:16K;3:16K;4:16K;5:16K;6:16K;7:16K;8:16K;9:16K;10:16K;11:16K;12:16K;13:16K;14:16K;15:16K;16:16K;17:16K;18:16K;19:16K;20:16K;21:16K;22:16K;23:16K;24:16K;25:16K;26:16K;27:16K;28:16K;29:16K;30:16K;31:16K;32:16K;33:16K;34:16K;35:16K;36:16K"
test
c23n12$ getfattr -n user.pvfs2.dist_params test# file: test
user.pvfs2.dist_params="strips:0:16K;1:16K;2:16K;3:16K;4:16K;5:16K;6:16K;7:16K;8:16K;9:16K;10:16K;11:16K;12:16K;13:16K;14:16K;15:16K;16:16K;17:16K;18:16K;19:16K;20:16K;21:16K;22:16K;23:16K;24:16K;25:16K;26:16K;27:16K;28:16K;29:16K;30:16K;31:16K;32:16K;33:16K;34:16K;35:16K;36:16K"
c23n12$ setfattr -n "user.pvfs2.dist_params" -v "strips:0:16K" test
c23n12$ getfattr -n user.pvfs2.dist_params test
# file: test
user.pvfs2.dist_params="strips:0:16K"
c23n12$ setfattr -n "user.pvfs2.dist_params" -v
"strips:0:16K;1:16K;2:16K;3:16K;4:16K;5:16K;6:16K;7:16K;8:16K;9:16K;10:16K;11:16K;12:16K;13:16K;14:16K;15:16K;16:16K;17:16K;18:16K;19:16K;20:16K;21:16K;22:16K;23:16K;24:16K;25:16K;26:16K;27:16K;28:16K;29:16K;30:16K;31:16K;32:16K;33:16K;34:16K;35:16K;36:16K;37:16K"
test
c23n12$ getfattr -n user.pvfs2.dist_params test# file: test
user.pvfs2.dist_params="strips:0:16K;1:16K;2:16K;3:16K;4:16K;5:16K;6:16K;7:16K;8:16K;9:16K;10:16K;11:16K;12:16K;13:16K;14:16K;15:16K;16:16K;17:16K;18:16K;19:16K;20:16K;21:16K;22:16K;23:16K;24:16K;25:16K;26:16K;27:16K;28:16K;29:16K;30:16K;31:16K;32:16K;33:16K;34:16K;35:16K;36:16K;37:16K"
c23n12$ setfattr -n "user.pvfs2.dist_params" -v
"strips:0:16K;1:16K;2:16K;3:16K;4:16K;5:16K;6:16K;7:16K;8:16K;9:16K;10:16K;11:16K;12:16K;13:16K;14:16K;15:16K;16:16K;17:16K;18:16K;19:16K;20:16K;21:16K;22:16K;23:16K;24:16K;25:16K;26:16K;27:16K;28:16K;29:16K;30:16K;31:16K;32:16K;33:16K;34:16K;35:16K;36:16K;37:16K"
test
c23n12$ getfattr -n user.pvfs2.dist_params test# file: test
user.pvfs2.dist_params="strips:0:16K;1:16K;2:16K;3:16K;4:16K;5:16K;6:16K;7:16K;8:16K;9:16K;10:16K;11:16K;12:16K;13:16K;14:16K;15:16K;16:16K;17:16K;18:16K;19:16K;20:16K;21:16K;22:16K;23:16K;24:16K;25:16K;26:16K;27:16K;28:16K;29:16K;30:16K;31:16K;32:16K;33:16K;34:16K;35:16K;36:16K;37:16K"
c23n12$ setfattr -n "user.pvfs2.dist_params" -v "strips:0:16K"
testc23n12$ getfattr -n user.pvfs2.dist_params test
# file: test
user.pvfs2.dist_params="strips:0:16K"
c23n12$ setfattr -n "user.pvfs2.dist_params" -v
"strips:0:16K;1:16K;2:16K;3:16K;4:16K;5:16K;6:16K;7:16K;8:16K;9:16K;10:16K;11:16K;12:16K;13:16K;14:16K;15:16K;16:16K;17:16K;18:16K;19:16K;20:16K;21:16K;22:16K;23:16K;24:16K;25:16K;26:16K;27:16K;28:16K;29:16K;30:16K;31:16K;32:16K;33:16K;34:16K;35:16K;36:16K;37:16K;38:16K;39:16K;40:16K;41:16K;42:16K;43:16K;44:16K;45:16K;46:16K;47:16K;48:16K;49:16K;50:16K;51:16K;52:16K;53:16K;54:16K;55:16K;56:16K;57:16K;58:16K;59:16K;60:16K;61:16K;62:16K;63:16K"
test
c23n12$ getfattr -n user.pvfs2.dist_params test# file: test
user.pvfs2.dist_params="strips:0:16K;1:16K;2:16K;3:16K;4:16K;5:16K;6:16K;7:16K;8:16K;9:16K;10:16K;11:16K;12:16K;13:16K;14:16K;15:16K;16:16K;17:16K;18:16K;19:16K;20:16K;21:16K;22:16K;23:16K;24:16K;25:16K;26:16K;27:16K;28:16K;29:16K;30:16K;31:16K;32:16K;33:16K;34:16K;35:16K;36:16K;37:16K;38:16K;39:16K;40:16K;41:16K;42:16K;43:16K;44:16K;45:16K;46:16K;47:16K;48:16K;49:16K;50:16K;51:16K;52:16K;53:16K;54:16K;55:16K;56:16K;57:16K;58:16K;59:16K;60:16K;61:16K;62:16K;63:16K"
c23n12$ setfattr -n "user.pvfs2.dist_params" -v "strips:0:16K" test
c23n12$ getfattr -n user.pvfs2.dist_params test
# file: test
user.pvfs2.dist_params="strips:0:16K"
c23n12$
I tried changing PVFS_MAX_XATTR_VALUELEN in include/pvfs2-types.h from
256 to 512 (only, rather than 16K) - still has problems , setting the
"user.pvfs2.dist_params"
param with a attribute value >256 characters works the first time you
try it, but no value
( irrespective of how ling it is) can be set on the directory
subsequently...
$ mount | grep /var/tmp/PVFSv2/mnt
tcp://ramones:3334/pvfs2-fs-kew on /var/tmp/PVFSv2/mnt type pvfs2 (rw)
$ cd /var/tmp/PVFSv2/mnt
$ mkdir test
$ setfattr -n "user.pvfs2.dist_name" -v "varstrip_dist" test
$ getfattr test
# file: test
user.pvfs2.dist_name
$ setfattr -n "user.pvfs2.dist_params" -v
"strips:0:16K;1:16K;2:16K;3:16K;4:16K;5:16K;6:16K;7:16K;8:16K;9:16K;10:16K;11:16K;12:16K;13:16K;14:16K;15:16K;16:16K;17:16K;18:16K;19:16K;20:16K;21:16K;22:16K;23:16K;24:16K;25:16K;26:16K;27:16K;28:16K;29:16K;30:16K;31:16K;32:16K;33:16K;34:16K;35:16K;36:16K"
test
$ getfattr test
# file: test
user.pvfs2.dist_name
user.pvfs2.dist_params
$ getfattr -n user.pvfs2.dist_params test
# file: test
user.pvfs2.dist_params="strips:0:16K;1:16K;2:16K;3:16K;4:16K;5:16K;6:16K;7:16K;8:16K;9:16K;10:16K;11:16K;12:16K;13:16K;14:16K;15:16K;16:16K;17:16K;18:16K;19:16K;20:16K;21:16K;22:16K;23:16K;24:16K;25:16K;26:16K;27:16K;28:16K;29:16K;30:16K;31:16K;32:16K;33:16K;34:16K;35:16K;36:16K"
$ setfattr -n "user.pvfs2.dist_params" -v "strips:0:16K" test
$ getfattr -n user.pvfs2.dist_params test
# file: test
user.pvfs2.dist_params="strips:0:16K"
$ setfattr -n "user.pvfs2.dist_params" -v
"strips:0:16K;1:16K;2:16K;3:16K;4:16K;5:16K;6:16K;7:16K;8:16K;9:16K;10:16K;11:16K;12:16K;13:16K;14:16K;15:16K;16:16K;17:16K;18:16K;19:16K;20:16K;21:16K;22:16K;23:16K;24:16K;25:16K;26:16K;27:16K;28:16K;29:16K;30:16K;31:16K;32:16K;33:16K;34:16K;35:16K;36:16K;37:16K"
test
$ getfattr -n user.pvfs2.dist_params test
# file: test
user.pvfs2.dist_params="strips:0:16K;1:16K;2:16K;3:16K;4:16K;5:16K;6:16K;7:16K;8:16K;9:16K;10:16K;11:16K;12:16K;13:16K;14:16K;15:16K;16:16K;17:16K;18:16K;19:16K;20:16K;21:16K;22:16K;23:16K;24:16K;25:16K;26:16K;27:16K;28:16K;29:16K;30:16K;31:16K;32:16K;33:16K;34:16K;35:16K;36:16K;37:16K"
$ setfattr -n "user.pvfs2.dist_params" -v "strips:0:16K"
testsetfattr: test: Invalid argument
$
Thanks,
Tony
Tony Kew
SAN Administrator
The Center for Computational Research
New York State Center of Excellence
in Bioinformatics & Life Sciences
701 Ellicott Street, Buffalo, NY 14203
CoE Office: (716) 881-8930 Fax: (716) 849-6656
CSE Office: (716) 645-3797 x2174
Cell: (716) 560-0910 Home: (716) 874-2126
"I love deadlines, I love the whooshing noise they make as they go by."
Douglas Adams
Phil Carns wrote:
Tony Kew wrote:
Dear Phil,
I can't test the setfattr command on a local filesystem as far as I
can tell:
$ cd /var/tmp
$ mkdir test
$ setfattr -n "user.pvfs2.dist_name" -v "varstrip_dist" test
setfattr: test: Operation not supported
$
...or is there a test I can do on an ext3 filesystem dir?
I changed the value in include/pvfs2-types.h to 16K (should be plenty
big enough I guess...) & recompiled - now setfattr works the first
time,
but fails subsequently..
Note: testing on a single RHEL5 node, if that makes a difference
<cut>
It is not easy for me to test this code on 64 nodes.
I don't know if we can go further with this without me doing so?
Well, it looks like there are still problems just setting the
attributes, so there isn't really any need to try testing it on 64
nodes until that much is fixed.
Regarding how to test on ext3, you will need to follow Emmanuel
Florac's suggestion about the mount option to use for ext3. I just
want to make sure there isn't a system limit on the attribute size
before trying to bump it up in PVFS.
16K may be too large for the request messaging system. If you
experiment with that it might be safer to pick 8K or smaller.
-Phil
_______________________________________________
Pvfs2-users mailing list
[email protected]
http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users
