Whoops, thanks for catching that. This additional patch should fix it.
thanks,
-Phil
Tony Kew wrote:
Dear Phil,
The patch works under Red Hat Enterprise Linux 5.2, but not under
RHEL 4 update 5, which doesn't have DB_BUFFER_SMALL in
/usr/include/db4/db.h
From RHEL5.2 /usr/include/db4/db.h
/*
* DB (user visible) error return codes.
*
* !!!
* For source compatibility with DB 2.X deadlock return (EAGAIN), use the
* following:
* #include <errno.h>
* #define DB_LOCK_DEADLOCK EAGAIN
*
* !!!
* We don't want our error returns to conflict with other packages where
* possible, so pick a base error value that's hopefully not common. We
* document that we own the error name space from -30,800 to -30,999.
*/
/* DB (public) error return codes. */
#define DB_BUFFER_SMALL (-30999)/* User memory too small for
return. */
#define DB_DONOTINDEX (-30998)/* "Null" return from 2ndary
callbk. */
#define DB_KEYEMPTY (-30997)/* Key/data deleted or never
created. */
#define DB_KEYEXIST (-30996)/* The key/data pair already
exists. */
#define DB_LOCK_DEADLOCK (-30995)/* Deadlock. */
#define DB_LOCK_NOTGRANTED (-30994)/* Lock unavailable. */
#define DB_LOG_BUFFER_FULL (-30993)/* In-memory log buffer full. */
#define DB_NOSERVER (-30992)/* Server panic return. */
#define DB_NOSERVER_HOME (-30991)/* Bad home sent to server. */
#define DB_NOSERVER_ID (-30990)/* Bad ID sent to server. */
#define DB_NOTFOUND (-30989)/* Key/data pair not found
(EOF). */
#define DB_OLD_VERSION (-30988)/* Out-of-date version. */
#define DB_PAGE_NOTFOUND (-30987)/* Requested page not found. */
#define DB_REP_DUPMASTER (-30986)/* There are two masters. */
#define DB_REP_HANDLE_DEAD (-30985)/* Rolled back a commit. */
#define DB_REP_HOLDELECTION (-30984)/* Time to hold an election. */
#define DB_REP_ISPERM (-30983)/* Cached not written perm
written.*/
#define DB_REP_NEWMASTER (-30982)/* We have learned of a new
master. */
#define DB_REP_NEWSITE (-30981)/* New site entered system. */
#define DB_REP_NOTPERM (-30980)/* Permanent log record not
written. */
#define DB_REP_STARTUPDONE (-30979)/* Client startup complete. */
#define DB_REP_UNAVAIL (-30978)/* Site cannot currently be
reached. */
#define DB_RUNRECOVERY (-30977)/* Panic return. */
#define DB_SECONDARY_BAD (-30976)/* Secondary index corrupt. */
#define DB_VERIFY_BAD (-30975)/* Verify failed; bad format. */
#define DB_VERSION_MISMATCH (-30974)/* Environment version mismatch. */
from RHEL 4 update 5 /usr/include/db4/db.h
/*
* DB (user visible) error return codes.
*
* !!!
* For source compatibility with DB 2.X deadlock return (EAGAIN), use the
* following:
* #include <errno.h>
* #define DB_LOCK_DEADLOCK EAGAIN
*
* !!!
* We don't want our error returns to conflict with other packages where
* possible, so pick a base error value that's hopefully not common. We
* document that we own the error name space from -30,800 to -30,999.
*/
/* DB (public) error return codes. */
#define DB_DONOTINDEX (-30999)/* "Null" return from 2ndary
callbk. */
#define DB_FILEOPEN (-30998)/* Rename/remove while file is
open. */
#define DB_KEYEMPTY (-30997)/* Key/data deleted or never
created. */
#define DB_KEYEXIST (-30996)/* The key/data pair already
exists. */
#define DB_LOCK_DEADLOCK (-30995)/* Deadlock. */
#define DB_LOCK_NOTGRANTED (-30994)/* Lock unavailable. */
#define DB_NOSERVER (-30993)/* Server panic return. */
#define DB_NOSERVER_HOME (-30992)/* Bad home sent to server. */
#define DB_NOSERVER_ID (-30991)/* Bad ID sent to server. */
#define DB_NOTFOUND (-30990)/* Key/data pair not found
(EOF). */
#define DB_OLD_VERSION (-30989)/* Out-of-date version. */
#define DB_PAGE_NOTFOUND (-30988)/* Requested page not found. */
#define DB_REP_DUPMASTER (-30987)/* There are two masters. */
#define DB_REP_HANDLE_DEAD (-30986)/* Rolled back a commit. */
#define DB_REP_HOLDELECTION (-30985)/* Time to hold an election. */
#define DB_REP_ISPERM (-30984)/* Cached not written perm
written.*/
#define DB_REP_NEWMASTER (-30983)/* We have learned of a new
master. */
#define DB_REP_NEWSITE (-30982)/* New site entered system. */
#define DB_REP_NOTPERM (-30981)/* Permanent log record not
written. */
#define DB_REP_OUTDATED (-30980)/* Site is too far behind
master. */
#define DB_REP_UNAVAIL (-30979)/* Site cannot currently be
reached. */
#define DB_RUNRECOVERY (-30978)/* Panic return. */
#define DB_SECONDARY_BAD (-30977)/* Secondary index corrupt. */
#define DB_VERIFY_BAD (-30976)/* Verify failed; bad format. */
Ideas?
Thanks Much,
Tony
Tony Kew
SAN Administrator
The Center for Computational Research
New York State Center of Excellence
in Bioinformatics & Life Sciences
701 Ellicott Street, Buffalo, NY 14203
CoE Office: (716) 881-8930 Fax: (716) 849-6656
CSE Office: (716) 645-3797 x2174
Cell: (716) 560-0910 Home: (716) 874-2126
"I love deadlines, I love the whooshing noise they make as they go by."
Douglas Adams
Tony Kew wrote:
Dear Phil,
The patch looks good - I can set a 64 node config now:
e.g.
ramones$ setfattr -n "user.pvfs2.dist_name" -v "varstrip_dist" test
ramones$ setfattr -n "user.pvfs2.dist_params" -v
"strips:0:16K;1:16K;2:16K;3:16K;4:16K;5:16K;6:16K;7:16K;8:16K;9:16K;10:16K;11:16K;12:16K;13:16K;14:16K;15:16K;16:16K;17:16K;18:16K;19:16K;20:16K;21:16K;22:16K;23:16K;24:16K;25:16K;26:16K;27:16K;28:16K;29:16K;30:16K;31:16K;32:16K;33:16K;34:16K;35:16K;36:16K;37:16K;38:16K;39:16K;40:16K;41:16K;42:16K;43:16K;44:16K;45:16K;46:16K;47:16K;48:16K;49:16K;50:16K;51:16K;52:16K;53:16K;54:16K;55:16K;56:16K;57:16K;58:16K;59:16K;60:16K;61:16K;62:16K;63:16K"
test
ramones$ getfattr -n "user.pvfs2.dist_params" test
# file: test
user.pvfs2.dist_params="strips:0:16K;1:16K;2:16K;3:16K;4:16K;5:16K;6:16K;7:16K;8:16K;9:16K;10:16K;11:16K;12:16K;13:16K;14:16K;15:16K;16:16K;17:16K;18:16K;19:16K;20:16K;21:16K;22:16K;23:16K;24:16K;25:16K;26:16K;27:16K;28:16K;29:16K;30:16K;31:16K;32:16K;33:16K;34:16K;35:16K;36:16K;37:16K;38:16K;39:16K;40:16K;41:16K;42:16K;43:16K;44:16K;45:16K;46:16K;47:16K;48:16K;49:16K;50:16K;51:16K;52:16K;53:16K;54:16K;55:16K;56:16K;57:16K;58:16K;59:16K;60:16K;61:16K;62:16K;63:16K"
ramones$
It may take a little while till I can install this on the cluster &
test PVFSv2
over 64 nodes, but at least the parameter can be set :-)
Thanks,
Tony
Tony Kew
SAN Administrator
The Center for Computational Research
New York State Center of Excellence
in Bioinformatics & Life Sciences
701 Ellicott Street, Buffalo, NY 14203
CoE Office: (716) 881-8930 Fax: (716) 849-6656
CSE Office: (716) 645-3797 x2174
Cell: (716) 560-0910 Home: (716) 874-2126
"I love deadlines, I love the whooshing noise they make as they go by."
Douglas Adams
Phil Carns wrote:
Hi Tony,
Would you mind trying the attached patch? It increases the maximum
xattr value size to 8KB and also fixes a problem further down in the
code regarding how to handle xattrs that large. The latter part of
the patch should solve the problem that you ran into in which you
could only set a large xattr once.
thanks,
-Phil
Tony Kew wrote:
Dear Phil,
I verified that this is not an issue with the underlying filesystem:
c23n12$ mount | grep \ /scratch\
/dev/sda3 on /scratch type ext2 (rw)
c23n12$ sudo umount /scratch
c23n12$ sudo mount -o rw,user_xattr /dev/sda3 /scratch
c23n12$ mount | grep \ /scratch\
/dev/sda3 on /scratch type ext2 (rw,user_xattr)
c23n12$ cd /scratch
c23n12$ mkdir test
c23n12$ setfattr -n "user.pvfs2.dist_name" -v "varstrip_dist" test
c23n12$ getfattr test/
# file: test
user.pvfs2.dist_name
c23n12$ setfattr -n "user.pvfs2.dist_params" -v
"strips:0:16K;1:16K;2:16K;3:16K;4:16K;5:16K;6:16K;7:16K;8:16K;9:16K;10:16K;11:16K;12:16K;13:16K;14:16K;15:16K;16:16K;17:16K;18:16K;19:16K;20:16K;21:16K;22:16K;23:16K;24:16K;25:16K;26:16K;27:16K;28:16K;29:16K;30:16K;31:16K;32:16K;33:16K;34:16K;35:16K;36:16K"
test
c23n12$ getfattr -n user.pvfs2.dist_params test# file: test
user.pvfs2.dist_params="strips:0:16K;1:16K;2:16K;3:16K;4:16K;5:16K;6:16K;7:16K;8:16K;9:16K;10:16K;11:16K;12:16K;13:16K;14:16K;15:16K;16:16K;17:16K;18:16K;19:16K;20:16K;21:16K;22:16K;23:16K;24:16K;25:16K;26:16K;27:16K;28:16K;29:16K;30:16K;31:16K;32:16K;33:16K;34:16K;35:16K;36:16K"
c23n12$ setfattr -n "user.pvfs2.dist_params" -v "strips:0:16K" test
c23n12$ getfattr -n user.pvfs2.dist_params test
# file: test
user.pvfs2.dist_params="strips:0:16K"
c23n12$ setfattr -n "user.pvfs2.dist_params" -v
"strips:0:16K;1:16K;2:16K;3:16K;4:16K;5:16K;6:16K;7:16K;8:16K;9:16K;10:16K;11:16K;12:16K;13:16K;14:16K;15:16K;16:16K;17:16K;18:16K;19:16K;20:16K;21:16K;22:16K;23:16K;24:16K;25:16K;26:16K;27:16K;28:16K;29:16K;30:16K;31:16K;32:16K;33:16K;34:16K;35:16K;36:16K;37:16K"
test
c23n12$ getfattr -n user.pvfs2.dist_params test# file: test
user.pvfs2.dist_params="strips:0:16K;1:16K;2:16K;3:16K;4:16K;5:16K;6:16K;7:16K;8:16K;9:16K;10:16K;11:16K;12:16K;13:16K;14:16K;15:16K;16:16K;17:16K;18:16K;19:16K;20:16K;21:16K;22:16K;23:16K;24:16K;25:16K;26:16K;27:16K;28:16K;29:16K;30:16K;31:16K;32:16K;33:16K;34:16K;35:16K;36:16K;37:16K"
c23n12$ setfattr -n "user.pvfs2.dist_params" -v
"strips:0:16K;1:16K;2:16K;3:16K;4:16K;5:16K;6:16K;7:16K;8:16K;9:16K;10:16K;11:16K;12:16K;13:16K;14:16K;15:16K;16:16K;17:16K;18:16K;19:16K;20:16K;21:16K;22:16K;23:16K;24:16K;25:16K;26:16K;27:16K;28:16K;29:16K;30:16K;31:16K;32:16K;33:16K;34:16K;35:16K;36:16K;37:16K"
test
c23n12$ getfattr -n user.pvfs2.dist_params test# file: test
user.pvfs2.dist_params="strips:0:16K;1:16K;2:16K;3:16K;4:16K;5:16K;6:16K;7:16K;8:16K;9:16K;10:16K;11:16K;12:16K;13:16K;14:16K;15:16K;16:16K;17:16K;18:16K;19:16K;20:16K;21:16K;22:16K;23:16K;24:16K;25:16K;26:16K;27:16K;28:16K;29:16K;30:16K;31:16K;32:16K;33:16K;34:16K;35:16K;36:16K;37:16K"
c23n12$ setfattr -n "user.pvfs2.dist_params" -v "strips:0:16K"
testc23n12$ getfattr -n user.pvfs2.dist_params test
# file: test
user.pvfs2.dist_params="strips:0:16K"
c23n12$ setfattr -n "user.pvfs2.dist_params" -v
"strips:0:16K;1:16K;2:16K;3:16K;4:16K;5:16K;6:16K;7:16K;8:16K;9:16K;10:16K;11:16K;12:16K;13:16K;14:16K;15:16K;16:16K;17:16K;18:16K;19:16K;20:16K;21:16K;22:16K;23:16K;24:16K;25:16K;26:16K;27:16K;28:16K;29:16K;30:16K;31:16K;32:16K;33:16K;34:16K;35:16K;36:16K;37:16K;38:16K;39:16K;40:16K;41:16K;42:16K;43:16K;44:16K;45:16K;46:16K;47:16K;48:16K;49:16K;50:16K;51:16K;52:16K;53:16K;54:16K;55:16K;56:16K;57:16K;58:16K;59:16K;60:16K;61:16K;62:16K;63:16K"
test
c23n12$ getfattr -n user.pvfs2.dist_params test# file: test
user.pvfs2.dist_params="strips:0:16K;1:16K;2:16K;3:16K;4:16K;5:16K;6:16K;7:16K;8:16K;9:16K;10:16K;11:16K;12:16K;13:16K;14:16K;15:16K;16:16K;17:16K;18:16K;19:16K;20:16K;21:16K;22:16K;23:16K;24:16K;25:16K;26:16K;27:16K;28:16K;29:16K;30:16K;31:16K;32:16K;33:16K;34:16K;35:16K;36:16K;37:16K;38:16K;39:16K;40:16K;41:16K;42:16K;43:16K;44:16K;45:16K;46:16K;47:16K;48:16K;49:16K;50:16K;51:16K;52:16K;53:16K;54:16K;55:16K;56:16K;57:16K;58:16K;59:16K;60:16K;61:16K;62:16K;63:16K"
c23n12$ setfattr -n "user.pvfs2.dist_params" -v "strips:0:16K" test
c23n12$ getfattr -n user.pvfs2.dist_params test
# file: test
user.pvfs2.dist_params="strips:0:16K"
c23n12$
I tried changing PVFS_MAX_XATTR_VALUELEN in include/pvfs2-types.h from
256 to 512 (only, rather than 16K) - still has problems , setting
the "user.pvfs2.dist_params"
param with a attribute value >256 characters works the first time
you try it, but no value
( irrespective of how ling it is) can be set on the directory
subsequently...
$ mount | grep /var/tmp/PVFSv2/mnt
tcp://ramones:3334/pvfs2-fs-kew on /var/tmp/PVFSv2/mnt type pvfs2 (rw)
$ cd /var/tmp/PVFSv2/mnt
$ mkdir test
$ setfattr -n "user.pvfs2.dist_name" -v "varstrip_dist" test
$ getfattr test
# file: test
user.pvfs2.dist_name
$ setfattr -n "user.pvfs2.dist_params" -v
"strips:0:16K;1:16K;2:16K;3:16K;4:16K;5:16K;6:16K;7:16K;8:16K;9:16K;10:16K;11:16K;12:16K;13:16K;14:16K;15:16K;16:16K;17:16K;18:16K;19:16K;20:16K;21:16K;22:16K;23:16K;24:16K;25:16K;26:16K;27:16K;28:16K;29:16K;30:16K;31:16K;32:16K;33:16K;34:16K;35:16K;36:16K"
test
$ getfattr test
# file: test
user.pvfs2.dist_name
user.pvfs2.dist_params
$ getfattr -n user.pvfs2.dist_params test
# file: test
user.pvfs2.dist_params="strips:0:16K;1:16K;2:16K;3:16K;4:16K;5:16K;6:16K;7:16K;8:16K;9:16K;10:16K;11:16K;12:16K;13:16K;14:16K;15:16K;16:16K;17:16K;18:16K;19:16K;20:16K;21:16K;22:16K;23:16K;24:16K;25:16K;26:16K;27:16K;28:16K;29:16K;30:16K;31:16K;32:16K;33:16K;34:16K;35:16K;36:16K"
$ setfattr -n "user.pvfs2.dist_params" -v "strips:0:16K" test
$ getfattr -n user.pvfs2.dist_params test
# file: test
user.pvfs2.dist_params="strips:0:16K"
$ setfattr -n "user.pvfs2.dist_params" -v
"strips:0:16K;1:16K;2:16K;3:16K;4:16K;5:16K;6:16K;7:16K;8:16K;9:16K;10:16K;11:16K;12:16K;13:16K;14:16K;15:16K;16:16K;17:16K;18:16K;19:16K;20:16K;21:16K;22:16K;23:16K;24:16K;25:16K;26:16K;27:16K;28:16K;29:16K;30:16K;31:16K;32:16K;33:16K;34:16K;35:16K;36:16K;37:16K"
test
$ getfattr -n user.pvfs2.dist_params test
# file: test
user.pvfs2.dist_params="strips:0:16K;1:16K;2:16K;3:16K;4:16K;5:16K;6:16K;7:16K;8:16K;9:16K;10:16K;11:16K;12:16K;13:16K;14:16K;15:16K;16:16K;17:16K;18:16K;19:16K;20:16K;21:16K;22:16K;23:16K;24:16K;25:16K;26:16K;27:16K;28:16K;29:16K;30:16K;31:16K;32:16K;33:16K;34:16K;35:16K;36:16K;37:16K"
$ setfattr -n "user.pvfs2.dist_params" -v "strips:0:16K"
testsetfattr: test: Invalid argument
$
Thanks,
Tony
Tony Kew
SAN Administrator
The Center for Computational Research
New York State Center of Excellence
in Bioinformatics & Life Sciences
701 Ellicott Street, Buffalo, NY 14203
CoE Office: (716) 881-8930 Fax: (716) 849-6656
CSE Office: (716) 645-3797 x2174
Cell: (716) 560-0910 Home: (716) 874-2126
"I love deadlines, I love the whooshing noise they make as they go by."
Douglas Adams
Phil Carns wrote:
Tony Kew wrote:
Dear Phil,
I can't test the setfattr command on a local filesystem as far as
I can tell:
$ cd /var/tmp
$ mkdir test
$ setfattr -n "user.pvfs2.dist_name" -v "varstrip_dist" test
setfattr: test: Operation not supported
$
...or is there a test I can do on an ext3 filesystem dir?
I changed the value in include/pvfs2-types.h to 16K (should be plenty
big enough I guess...) & recompiled - now setfattr works the
first time,
but fails subsequently..
Note: testing on a single RHEL5 node, if that makes a difference
<cut>
It is not easy for me to test this code on 64 nodes.
I don't know if we can go further with this without me doing so?
Well, it looks like there are still problems just setting the
attributes, so there isn't really any need to try testing it on 64
nodes until that much is fixed.
Regarding how to test on ext3, you will need to follow Emmanuel
Florac's suggestion about the mount option to use for ext3. I just
want to make sure there isn't a system limit on the attribute size
before trying to bump it up in PVFS.
16K may be too large for the request messaging system. If you
experiment with that it might be safer to pick 8K or smaller.
-Phil
---------------------
PatchSet 7171
Date: 2008/10/22 17:44:41
Author: pcarns
Branch: HEAD
Tag: (none)
Log:
adjust recent keyval_write_list() fix to be compatible with older db
libraries
Members:
src/io/trove/trove-dbpf/dbpf-keyval.c:1.90->1.91
Index: pvfs2-1/src/io/trove/trove-dbpf/dbpf-keyval.c
diff -u pvfs2-1/src/io/trove/trove-dbpf/dbpf-keyval.c:1.90 pvfs2-1/src/io/trove/trove-dbpf/dbpf-keyval.c:1.91
--- pvfs2-1/src/io/trove/trove-dbpf/dbpf-keyval.c:1.90 Wed Oct 22 12:02:47 2008
+++ pvfs2-1/src/io/trove/trove-dbpf/dbpf-keyval.c Wed Oct 22 17:44:41 2008
@@ -1258,7 +1258,11 @@
/* check for DB_BUFFER_SMALL in case the key is there but the data
* is simply too big for the temporary data buffer used
*/
+#ifdef HAVE_DB_BUFFER_SMALL
if (ret != 0 && ret != DB_BUFFER_SMALL)
+#else
+ if (ret != 0 && ret != ENOMEM)
+#endif
{
if(ret == DB_NOTFOUND && ((op_p->flags & TROVE_NOOVERWRITE) ||
(!(op_p->flags & TROVE_ONLYOVERWRITE))))
_______________________________________________
Pvfs2-users mailing list
[email protected]
http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users
