Yeah, that's a problem. Each of those databases uses a different sort
order, so you'll need to try again with a slightly different
modification to BDB. I updated the documentation in trunk and attached
the new version to this email. I haven't had an opportunity to test it
yet, but it's a pretty simple change.
thanks,
-Phil
Doug Johnson wrote:
The db_load program was modified using the instructions in
doc/db-recovery.txt given in the latest pvfs release. Should I be
worried about the following lines from this file?
The dataspace_attributes.db file requires a different modification
(not provided here).
Doug
At Fri, 19 Mar 2010 10:28:03 -0400,
Phil Carns wrote:
Oh, I guess that db_verify doesn't need to be modified if you are just
using the -o argument. db_load still has to be modified, though.
-Phil
Phil Carns wrote:
Hi Doug,
Did you modify all three of db_dump, db_load, and db_verify with the
comparison function that PVFS uses for that db?
-Phil
Doug Johnson wrote:
Hi Phil,
I probably should have tried the latest version first, I picked the
older version that matched what was installed on the system out of
an abundance of caution.
I've installed a modified db-4.8.26. There was some progress in the
dump/restore procedure,
opt2326:/fs/pvfs/pvfs/5810ab5d> type db_dump
db_dump is hashed (/usr/local/pvfs-db-4.8.26/bin/db_dump)
opt2326:/fs/pvfs/pvfs/5810ab5d> db_dump -r \
-f /tmp/dataspace_attributes.out \
dataspace_attributes.db
opt2326:/fs/pvfs/pvfs/5810ab5d> echo $?
0
opt2326:/fs/pvfs/pvfs/5810ab5d> db_load -f
/tmp/dataspace_attributes.out test.db
opt2326:/fs/pvfs/pvfs/5810ab5d> mv dataspace_attributes.db \
dataspace_attributes.db.bad
opt2326:/fs/pvfs/pvfs/5810ab5d> db_verify -o dataspace_attributes.db
opt2326:/fs/pvfs/pvfs/5810ab5d> echo $?
0
However, when attempting to start the pvfs server it exits immediately
with,
[D 03/18 17:02] PVFS2 Server version 2.8.1 starting.
[E 03/18 17:02] dbpf_dspace_iterate_handles_op_svc: Invalid argument
[E 03/18 17:02] Error adding handle range
4099276460824344803-5124095576030431002 to
filesystem pvfs2-fs
[E 03/18 17:02] Error: Could not initialize server interfaces; aborting.
[E 03/18 17:02] Error: Could not initialize server; aborting.
The logs with EventLogging set to 'all' are attached. I've rechecked
the bdb files, and all verify with no errors.
Doug
At Thu, 18 Mar 2010 15:08:52 -0400,
Phil Carns wrote:
Hi Doug,
I haven't seen db_dump fail altogether like that before. I can make
some suggestions that you may want to try, though. You should probably
back up the current corrupted db first to make sure things don't get
worse.
The first thing I would suggest is to try a newer version of berkeley db
for db_dump, since you are having to build that tool from scratch
anyway. Newer versions of bdb can read the same format, and maybe there
is a chance that the verification/recovery has improved since 4.3.
You can also try db_recover as well (perhaps with -c?). That tool is
really meant to recover transactions (which is not the issue here), but
maybe it can shed more light on the verification problem.
Finally in db_dump, you can try -R instead of -r. I would save that as
a last resort, because I think it may recover unwanted (bogus) data as
well.
-Phil
# 2009-11-5
# Notes on recoverying corrupted Berkeley DB files in PVFS
====================================================================
The pvfs2-server daemon uses Berkeley DB as the mechanism for storing file
system metadata. There are 5 database files in total that can be found in
the following locations in the storage space:
./storage_attributes.db
./50a6d673/collection_attributes.db
./50a6d673/dataspace_attributes.db
./50a6d673/keyval.db
./collections.db
The dataspace_attributes.db and keyval.db are most frequently used by
the file system. If one of these database files is corrupted for some
reason, then it may prevent the file system from operating correctly.
One common technique for repairing a Berkeley DB database is to dump its
contents using db_dump (possibly found in the db4-utils package) and then
reload it into a new .db file with db_load. However, both the keyval.db and
dataspace_attributes.db use a custom function for sorting entries in order
to improve PVFS performance. db_load must therefore be modified to use the
correct key order.
Here are the steps needed to build a db_load utility that will work on the
keyval.db or dataspace_attributes.db file:
- download the source code for Berkeley DB
- edit db_load/db_load.c
- find the section marked by #if 0 that indicates where to insert
application specific btree comparison or hash functions
- insert the code listed at the end of this file (NOTE: there is different
code depending on which .db file you are trying to recover)
- build Berkeley DB
- rename db_load binary to db_load_pvfs_keyval to avoid confusion
For keyval.db:
====================================================================
#include <stdint.h>
typedef uint64_t PVFS_handle;
typedef PVFS_handle TROVE_handle;
#define PVFS_NAME_MAX 256
#define DBPF_MAX_KEY_LENGTH PVFS_NAME_MAX
struct dbpf_keyval_db_entry
{
TROVE_handle handle;
char key[DBPF_MAX_KEY_LENGTH];
};
#define DBPF_KEYVAL_DB_ENTRY_TOTAL_SIZE(_size) \
(sizeof(TROVE_handle) + _size)
#define DBPF_KEYVAL_DB_ENTRY_KEY_SIZE(_size) \
(_size - sizeof(TROVE_handle))
int PINT_trove_dbpf_keyval_compare(
DB * dbp, const DBT * a, const DBT * b)
{
const struct dbpf_keyval_db_entry * db_entry_a;
const struct dbpf_keyval_db_entry * db_entry_b;
db_entry_a = (const struct dbpf_keyval_db_entry *) a->data;
db_entry_b = (const struct dbpf_keyval_db_entry *) b->data;
if(db_entry_a->handle != db_entry_b->handle)
{
return (db_entry_a->handle < db_entry_b->handle) ? -1 : 1;
}
if(a->size > b->size)
{
return 1;
}
if(a->size < b->size)
{
return -1;
}
/* must be equal */
return (memcmp(db_entry_a->key, db_entry_b->key,
DBPF_KEYVAL_DB_ENTRY_KEY_SIZE(a->size)));
}
if ((ret = dbp->set_bt_compare(dbp, PINT_trove_dbpf_keyval_compare)) != 0)
dbp->err(dbp, ret, "DB->set_bt_compare");
goto err;
}
====================================================================
For dataspace_attributes.db:
====================================================================
#include <stdint.h>
typedef uint64_t PVFS_handle;
typedef PVFS_handle TROVE_handle;
int PINT_trove_dbpf_ds_attr_compare(DB * dbp, const DBT * a, const DBT * b)
{
const TROVE_handle * handle_a;
const TROVE_handle * handle_b;
handle_a = (const TROVE_handle *) a->data;
handle_b = (const TROVE_handle *) b->data;
if(*handle_a == *handle_b)
{
return 0;
}
return (*handle_a > *handle_b) ? -1 : 1;
}
if ((ret = dbp->set_bt_compare(dbp, PINT_trove_dbpf_ds_attr_compare)) != 0) {
dbp->err(dbp, ret, "DB->set_bt_compare");
goto err;
}
====================================================================
_______________________________________________
Pvfs2-users mailing list
[email protected]
http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users
