Hi Tony,
Looking at the length of those strings, it looks like the problem starts
when the xattr value is 256 characters. Digging around in the code it
looks like we have a hard coded limit on this in PVFS (from pvfs2-types.h):
#define PVFS_MAX_XATTR_VALUELEN 256
I don't know if there is a good reason for this limit or if it is
arbitrary. Could you test and see if you can set an extended attribute
with that value on a local file system on your box?
I don't think we have had anyone try to specify varstrip parameters for
that many datafiles before :)
thanks,
-Phil
Tony Kew wrote:
Dear All,
I'm running pvfs 7.2.1 with Murali Vilayannur xarrts patch from this
message thread:
http://osdir.com/ml/file-systems.pvfs2.users/2007-01/msg00050.html
(which, if memory serves, Phil Carns put into CVS)
I have a PVFS filesystem over 64 nodes, mounted on all those nodes:
$ pvfs2-ping -m /scratch/1076995.bono.ccr.buffalo.edu/mnt
(1) Parsing tab file...
(2) Initializing system interface...
(3) Initializing each file system found in tab file: /etc/mtab...
PVFS2 servers: tcp://c27n20:3334
Storage name: pvfs2-fs-1076995.bono.ccr.buffalo.edu
Local mount point: /scratch/1076995.bono.ccr.buffalo.edu/mnt
/scratch/1076995.bono.ccr.buffalo.edu/mnt: Ok
(4) Searching for /scratch/1076995.bono.ccr.buffalo.edu/mnt in pvfstab...
PVFS2 servers: tcp://c27n20:3334
Storage name: pvfs2-fs-1076995.bono.ccr.buffalo.edu
Local mount point: /scratch/1076995.bono.ccr.buffalo.edu/mnt
meta servers:
tcp://c25n17:3334
tcp://c25n18:3334
tcp://c25n19:3334
tcp://c25n20:3334
tcp://c25n21:3334
tcp://c25n22:3334
tcp://c25n23:3334
tcp://c25n24:3334
tcp://c25n25:3334
tcp://c25n26:3334
tcp://c25n27:3334
tcp://c25n28:3334
tcp://c26n01:3334
tcp://c26n02:3334
tcp://c26n03:3334
tcp://c26n04:3334
tcp://c26n05:3334
tcp://c26n06:3334
tcp://c26n07:3334
tcp://c26n08:3334
tcp://c26n09:3334
tcp://c26n10:3334
tcp://c26n11:3334
tcp://c26n12:3334
tcp://c26n13:3334
tcp://c26n14:3334
tcp://c26n15:3334
tcp://c26n16:3334
tcp://c26n17:3334
tcp://c26n18:3334
tcp://c26n19:3334
tcp://c26n20:3334
tcp://c26n21:3334
tcp://c26n22:3334
tcp://c26n23:3334
tcp://c26n24:3334
tcp://c26n25:3334
tcp://c26n26:3334
tcp://c26n27:3334
tcp://c26n28:3334
tcp://c26n29:3334
tcp://c26n30:3334
tcp://c26n31:3334
tcp://c26n32:3334
tcp://c27n01:3334
tcp://c27n02:3334
tcp://c27n03:3334
tcp://c27n04:3334
tcp://c27n05:3334
tcp://c27n06:3334
tcp://c27n07:3334
tcp://c27n08:3334
tcp://c27n09:3334
tcp://c27n10:3334
tcp://c27n11:3334
tcp://c27n12:3334
tcp://c27n13:3334
tcp://c27n14:3334
tcp://c27n15:3334
tcp://c27n16:3334
tcp://c27n17:3334
tcp://c27n18:3334
tcp://c27n19:3334
tcp://c27n20:3334
data servers:
tcp://c25n17:3334
tcp://c25n18:3334
tcp://c25n19:3334
tcp://c25n20:3334
tcp://c25n21:3334
tcp://c25n22:3334
tcp://c25n23:3334
tcp://c25n24:3334
tcp://c25n25:3334
tcp://c25n26:3334
tcp://c25n27:3334
tcp://c25n28:3334
tcp://c26n01:3334
tcp://c26n02:3334
tcp://c26n03:3334
tcp://c26n04:3334
tcp://c26n05:3334
tcp://c26n06:3334
tcp://c26n07:3334
tcp://c26n08:3334
tcp://c26n09:3334
tcp://c26n10:3334
tcp://c26n11:3334
tcp://c26n12:3334
tcp://c26n13:3334
tcp://c26n14:3334
tcp://c26n15:3334
tcp://c26n16:3334
tcp://c26n17:3334
tcp://c26n18:3334
tcp://c26n19:3334
tcp://c26n20:3334
tcp://c26n21:3334
tcp://c26n22:3334
tcp://c26n23:3334
tcp://c26n24:3334
tcp://c26n25:3334
tcp://c26n26:3334
tcp://c26n27:3334
tcp://c26n28:3334
tcp://c26n29:3334
tcp://c26n30:3334
tcp://c26n31:3334
tcp://c26n32:3334
tcp://c27n01:3334
tcp://c27n02:3334
tcp://c27n03:3334
tcp://c27n04:3334
tcp://c27n05:3334
tcp://c27n06:3334
tcp://c27n07:3334
tcp://c27n08:3334
tcp://c27n09:3334
tcp://c27n10:3334
tcp://c27n11:3334
tcp://c27n12:3334
tcp://c27n13:3334
tcp://c27n14:3334
tcp://c27n15:3334
tcp://c27n16:3334
tcp://c27n17:3334
tcp://c27n18:3334
tcp://c27n19:3334
tcp://c27n20:3334
(5) Verifying that all servers are responding...
meta servers:
tcp://c25n17:3334 Ok
tcp://c25n18:3334 Ok
tcp://c25n19:3334 Ok
tcp://c25n20:3334 Ok
tcp://c25n21:3334 Ok
tcp://c25n22:3334 Ok
tcp://c25n23:3334 Ok
tcp://c25n24:3334 Ok
tcp://c25n25:3334 Ok
tcp://c25n26:3334 Ok
tcp://c25n27:3334 Ok
tcp://c25n28:3334 Ok
tcp://c26n01:3334 Ok
tcp://c26n02:3334 Ok
tcp://c26n03:3334 Ok
tcp://c26n04:3334 Ok
tcp://c26n05:3334 Ok
tcp://c26n06:3334 Ok
tcp://c26n07:3334 Ok
tcp://c26n08:3334 Ok
tcp://c26n09:3334 Ok
tcp://c26n10:3334 Ok
tcp://c26n11:3334 Ok
tcp://c26n12:3334 Ok
tcp://c26n13:3334 Ok
tcp://c26n14:3334 Ok
tcp://c26n15:3334 Ok
tcp://c26n16:3334 Ok
tcp://c26n17:3334 Ok
tcp://c26n18:3334 Ok
tcp://c26n19:3334 Ok
tcp://c26n20:3334 Ok
tcp://c26n21:3334 Ok
tcp://c26n22:3334 Ok
tcp://c26n23:3334 Ok
tcp://c26n24:3334 Ok
tcp://c26n25:3334 Ok
tcp://c26n26:3334 Ok
tcp://c26n27:3334 Ok
tcp://c26n28:3334 Ok
tcp://c26n29:3334 Ok
tcp://c26n30:3334 Ok
tcp://c26n31:3334 Ok
tcp://c26n32:3334 Ok
tcp://c27n01:3334 Ok
tcp://c27n02:3334 Ok
tcp://c27n03:3334 Ok
tcp://c27n04:3334 Ok
tcp://c27n05:3334 Ok
tcp://c27n06:3334 Ok
tcp://c27n07:3334 Ok
tcp://c27n08:3334 Ok
tcp://c27n09:3334 Ok
tcp://c27n10:3334 Ok
tcp://c27n11:3334 Ok
tcp://c27n12:3334 Ok
tcp://c27n13:3334 Ok
tcp://c27n14:3334 Ok
tcp://c27n15:3334 Ok
tcp://c27n16:3334 Ok
tcp://c27n17:3334 Ok
tcp://c27n18:3334 Ok
tcp://c27n19:3334 Ok
tcp://c27n20:3334 Ok
data servers:
tcp://c25n17:3334 Ok
tcp://c25n18:3334 Ok
tcp://c25n19:3334 Ok
tcp://c25n20:3334 Ok
tcp://c25n21:3334 Ok
tcp://c25n22:3334 Ok
tcp://c25n23:3334 Ok
tcp://c25n24:3334 Ok
tcp://c25n25:3334 Ok
tcp://c25n26:3334 Ok
tcp://c25n27:3334 Ok
tcp://c25n28:3334 Ok
tcp://c26n01:3334 Ok
tcp://c26n02:3334 Ok
tcp://c26n03:3334 Ok
tcp://c26n04:3334 Ok
tcp://c26n05:3334 Ok
tcp://c26n06:3334 Ok
tcp://c26n07:3334 Ok
tcp://c26n08:3334 Ok
tcp://c26n09:3334 Ok
tcp://c26n10:3334 Ok
tcp://c26n11:3334 Ok
tcp://c26n12:3334 Ok
tcp://c26n13:3334 Ok
tcp://c26n14:3334 Ok
tcp://c26n15:3334 Ok
tcp://c26n16:3334 Ok
tcp://c26n17:3334 Ok
tcp://c26n18:3334 Ok
tcp://c26n19:3334 Ok
tcp://c26n20:3334 Ok
tcp://c26n21:3334 Ok
tcp://c26n22:3334 Ok
tcp://c26n23:3334 Ok
tcp://c26n24:3334 Ok
tcp://c26n25:3334 Ok
tcp://c26n26:3334 Ok
tcp://c26n27:3334 Ok
tcp://c26n28:3334 Ok
tcp://c26n29:3334 Ok
tcp://c26n30:3334 Ok
tcp://c26n31:3334 Ok
tcp://c26n32:3334 Ok
tcp://c27n01:3334 Ok
tcp://c27n02:3334 Ok
tcp://c27n03:3334 Ok
tcp://c27n04:3334 Ok
tcp://c27n05:3334 Ok
tcp://c27n06:3334 Ok
tcp://c27n07:3334 Ok
tcp://c27n08:3334 Ok
tcp://c27n09:3334 Ok
tcp://c27n10:3334 Ok
tcp://c27n11:3334 Ok
tcp://c27n12:3334 Ok
tcp://c27n13:3334 Ok
tcp://c27n14:3334 Ok
tcp://c27n15:3334 Ok
tcp://c27n16:3334 Ok
tcp://c27n17:3334 Ok
tcp://c27n18:3334 Ok
tcp://c27n19:3334 Ok
tcp://c27n20:3334 Ok
(6) Verifying that fsid 1076995 is acceptable to all servers...
Ok; all servers understand fs_id 1076995
(7) Verifying that root handle is owned by one server...
Root handle: 1048576
Ok; root handle is owned by exactly one server.
=============================================================
The PVFS2 filesystem at /scratch/1076995.bono.ccr.buffalo.edu/mnt
appears to be correctly configured.
$
All nodes 64 nodes are up:
$ pvfs2-ping -m /scratch/1076995.bono.ccr.buffalo.edu/mnt | sed -n
'/^(5)/,/^(6)/p' | sed -n '/data servers/,/^$/p' | sed 1d | sed '$d' |
wc -l
64
$
I'm trying to create a varstrip_dist dir with (effectively) a stripe
across all 64 nodes,
with 16K on each node, but the setfattr fails for 37 (or more) nodes:
$ mount | grep pvfs2
tcp://c27n20:3334/pvfs2-fs-1076995.bono.ccr.buffalo.edu on
/scratch/1076995.bono.ccr.buffalo.edu/mnt type pvfs2 (rw)
$ cd /scratch/1076995.bono.ccr.buffalo.edu/mnt
$ mkdir test
$ setfattr -n "user.pvfs2.dist_name" -v "varstrip_dist" test
$ getfattr test
# file: test
user.pvfs2.dist_name
$ getfattr -n "user.pvfs2.dist_name" test
# file: test
user.pvfs2.dist_name="varstrip_dist"
$ setfattr -n "user.pvfs2.dist_params" -v
"strips:0:16K;1:16K;2:16K;3:16K;4:16K;5:16K;6:16K;7:16K;8:16K;9:16K;10:16K;11:16K;12:16K;13:16K;14:16K;15:16K;16:16K;17:16K;18:16K;19:16K;20:16K;21:16K;22:16K;23:16K;24:16K;25:16K;26:16K;27:16K;28:16K;29:16K;30:16K;31:16K;32:16K;33:16K;34:16K;35:16K;36:16K"
test
$ getfattr -n "user.pvfs2.dist_params" test
# file: test
user.pvfs2.dist_params="strips:0:16K;1:16K;2:16K;3:16K;4:16K;5:16K;6:16K;7:16K;8:16K;9:16K;10:16K;11:16K;12:16K;13:16K;14:16K;15:16K;16:16K;17:16K;18:16K;19:16K;20:16K;21:16K;22:16K;23:16K;24:16K;25:16K;26:16K;27:16K;28:16K;29:16K;30:16K;31:16K;32:16K;33:16K;34:16K;35:16K;36:16K"
$ setfattr -n "user.pvfs2.dist_params" -v
"strips:0:16K;1:16K;2:16K;3:16K;4:16K;5:16K;6:16K;7:16K;8:16K;9:16K;10:16K;11:16K;12:16K;13:16K;14:16K;15:16K;16:16K;17:16K;18:16K;19:16K;20:16K;21:16K;22:16K;23:16K;24:16K;25:16K;26:16K;27:16K;28:16K;29:16K;30:16K;31:16K;32:16K;33:16K;34:16K;35:16K;36:16K;37:16K"
test
setfattr: test: Invalid argument
$ setfattr -n "user.pvfs2.dist_params" -v
"strips:0:16K;1:16K;2:16K;3:16K;4:16K;5:16K;6:16K;7:16K;8:16K;9:16K;10:16K;11:16K;12:16K;13:16K;14:16K;15:16K;16:16K;17:16K;18:16K;19:16K;20:16K;21:16K;22:16K;23:16K;24:16K;25:16K;26:16K;27:16K;28:16K;29:16K;30:16K;31:16K;32:16K;33:16K;34:16K;35:16K;36:16K;37:16K;38:16K;39:16K;40:16K;41:16K;42:16K;43:16K;44:16K;45:16K;46:16K;47:16K;48:16K;49:16K;50:16K;51:16K;52:16K;53:16K;54:16K;55:16K;56:16K;57:16K;58:16K;59:16K;60:16K;61:16K;62:16K;63:16K"
test
setfattr: test: Invalid argument
$
Sorry for the over long lines (I made sure its not a problem with line
wrapping).
Since the error is from setfattr I wonder if this is a RedHat problem...
I'm running RedHat Enterprise Linux AS v.4 update 5 on these nodes, and
the attr package (which includes setfattr) is the latest version from RH:
$ rpm -qf `type -p setfattr`
attr-2.4.16-3.1.el4
$
Thanks Much,
Tony
Tony Kew
SAN Administrator
The Center for Computational Research
New York State Center of Excellence
in Bioinformatics & Life Sciences
701 Ellicott Street, Buffalo, NY 14203
CoE Office: (716) 881-8930 Fax: (716) 849-6656
CSE Office: (716) 645-3797 x2174
Cell: (716) 560-0910 Home: (716) 874-2126
"I love deadlines, I love the whooshing noise they make as they go by."
Douglas Adams
_______________________________________________
Pvfs2-users mailing list
[email protected]
http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users
_______________________________________________
Pvfs2-users mailing list
[email protected]
http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users
