I can't send email to freenetwork at web.de. The contact form to find out
why my mail was rejected is in German, even though the explanation page
has an english version, so frankly I can't be bothered. web.de sucks,
don't use it. And I *was* using my ISP's email server. Presumably either
a) they don't like me using a dyndns address, even though I'm using my
ISP as a smarthost (no reverse DNS rule, perhaps), or b) blueyonder
itself is marked as a spammer (which has happened in the past, and was
part of the reason I set up a local MTA in the first place!).
So I am continuing my conversation here, see below.
----- Forwarded message from Mail Delivery System <Mailer-Daemon at
blueyonder.co.uk> -----
X-Spam-Checker-Version: SpamAssassin 3.1.4 (2006-07-26) on servalan
X-Spam-Level:
X-Spam-Status: No, score=-1.7 required=5.0 tests=AWL,BAYES_00 autolearn=ham
version=3.1.4
Envelope-to: toad at amphibian.dyndns.org
Delivery-date: Sat, 11 Nov 2006 21:20:07 +0000
X-Failed-Recipients: freenetwork at web.de
Auto-Submitted: auto-generated
From: Mail Delivery System <mailer-dae...@blueyonder.co.uk>
To: toad at amphibian.dyndns.org
Subject: Mail delivery failed: returning message to sender
This message was created automatically by mail delivery software.
A message that you sent could not be delivered to one or more of its
recipients. This is a permanent error. The following address(es) failed:
freenetwork at web.de
SMTP error from remote mail server after initial connection:
host mx-ha01.web.de [217.72.192.149]: 554 Transaction failed. For
explanation visit http://freemail.web.de/reject/?ip=195.188.213.7
------ This is a copy of the message, including all the headers. ------
Return-path: <toad at amphibian.dyndns.org>
Received: from [172.23.170.138] (helo=anti-virus01-09)
by smtp-out4.blueyonder.co.uk with smtp (Exim 4.52)
id 1Gj0Gj-00019e-R6
for freenetwork at web.de; Sat, 11 Nov 2006 21:20:05 +0000
Received: from [82.32.17.1] (helo=localhost.localdomain)
by asmtp-out2.blueyonder.co.uk with esmtp (Exim 4.52)
id 1Gj0Gh-0002i6-Ee
for freenetwork at web.de; Sat, 11 Nov 2006 21:20:04 +0000
Received: from toad by localhost.localdomain with local (Exim 4.63)
(envelope-from <toad at amphibian.dyndns.org>)
id 1Gj0Gh-0002GN-6K
for freenetwork at web.de; Sat, 11 Nov 2006 21:20:03 +0000
Date: Sat, 11 Nov 2006 21:20:03 +0000
To: "freenetwork at web.de" <freenetwork at web.de>
Subject: Re: [freenet-cvs] r10888 - in trunk/freenet/src/freenet: node store
Message-ID: <20061111212003.GA8891 at amphibian.dyndns.org>
References: <20061111204655.06B0E9BCE4 at emu.freenetproject.org>
<E1Gj015-0001SC-00 at smtp06.web.de>
MIME-Version: 1.0
Content-Type: multipart/signed; micalg=pgp-sha1;
protocol="application/pgp-signature"; boundary="6c2NcOVqGQ03X4Wi"
Content-Disposition: inline
In-Reply-To: <E1Gj015-0001SC-00 at smtp06.web.de>
User-Agent: Mutt/1.5.13 (2006-08-11)
From: toad <t...@amphibian.dyndns.org>
--6c2NcOVqGQ03X4Wi
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable
I don't think that's a bug in the code I committed. Does it work anyway?
On Sat, Nov 11, 2006 at 10:01:09PM +0100, freenetwork at web.de wrote:
> [...]
> Migrating data from old Environment to new Environment
> <Cleaner name=3D"Cleaner-1"/> caught exception: java.lang.NullPointerExce=
ption
> java.lang.NullPointerException
> at java.util.TreeMap.getEntry(TreeMap.java:324)
> at java.util.TreeMap.remove(TreeMap.java:580)
> at java.util.TreeSet.remove(TreeSet.java:259)
> at com.sleepycat.je.cleaner.FileSelector.selectFileForCleaning(Fi=
leSelector.java:236)
> at com.sleepycat.je.cleaner.FileProcessor.doClean(FileProcessor.j=
ava:201)
> at com.sleepycat.je.cleaner.FileProcessor.onWakeup(FileProcessor.=
java:143)
> at com.sleepycat.je.utilint.DaemonThread.run(DaemonThread.java:18=
9)
> at java.lang.Thread.run(Thread.java:619)
> Continuing
> Completed migration.
> [...]
>=20
>=20
> >Author: toad
> >Date: 2006-11-11 20:46:52 +0000 (Sat, 11 Nov 2006)
> >New Revision: 10888
>=20
> >Modified:
> > trunk/freenet/src/freenet/node/Node.java
> > trunk/freenet/src/freenet/store/BerkeleyDBFreenetStore.java
> >Log:
> >Single Environment for all datastores, (in database-<port>), with store =
f=3D
> >iles moved to the storeDir.
> >Automatic migration.
> >Small gains in speed and memory usage expected.
>=20
> >Modified: trunk/freenet/src/freenet/node/Node.java
> >=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=
=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D
> >=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=
=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D
> >=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=
=3D3D=3D3D=3D3D
> >--- trunk/freenet/src/freenet/node/Node.java 2006-11-11 17:05:05 UTC (re=
v=3D
> > 10887)
> >+++ trunk/freenet/src/freenet/node/Node.java 2006-11-11 20:46:52 UTC (re=
v=3D
> > 10888)
> >@@ -1112,7 +1112,7 @@
> > =3D09
> > File dbDir =3D3D new File(storeDir, "database-"+portNumber);
> > dbDir.mkdirs();
> >-
> >+ =3D09
> > try {
> > storeEnvironment =3D3D new Environment(dbDir,
> > envConfig);
> > envMutableConfig =3D3D
> > storeEnvironment.getMutableConfig();
> >@@ -1143,29 +1143,29 @@
> > try {
> > Logger.normal(this, "Initializing CHK Datastore");
> > System.out.println("Initializing CHK Datastore
> > ("+maxStoreKeys+" key=
s=3D
> >)");
> >- chkDatastore =3D3D
> >BerkeleyDBFreenetStore.construct(lastVersion, "",=
st=3D
> >oreDir, true, suffix, maxStoreKeys,=3D20
> >- CHKBlock.DATA_LENGTH,
> >CHKBlock.TOTAL_HEADERS_LENGTH, true, Berkele=
y=3D
> >DBFreenetStore.TYPE_CHK);
> >+ chkDatastore =3D3D
> >BerkeleyDBFreenetStore.construct(lastVersion, sto=
reD=3D
> >ir, true, suffix, maxStoreKeys,=3D20
> >+ CHKBlock.DATA_LENGTH,
> >CHKBlock.TOTAL_HEADERS_LENGTH, true, Berkele=
y=3D
> >DBFreenetStore.TYPE_CHK, storeEnvironment, random);
> > Logger.normal(this, "Initializing CHK Datacache");
> > System.out.println("Initializing CHK Datacache
> > ("+maxCacheKeys+":"+m=
a=3D
> >xCacheKeys+" keys)");
> >- chkDatacache =3D3D
> >BerkeleyDBFreenetStore.construct(lastVersion, "",=
st=3D
> >oreDir, false, suffix, maxCacheKeys,=3D20
> >- CHKBlock.DATA_LENGTH,
> >CHKBlock.TOTAL_HEADERS_LENGTH, true, Berkele=
y=3D
> >DBFreenetStore.TYPE_CHK);
> >+ chkDatacache =3D3D
> >BerkeleyDBFreenetStore.construct(lastVersion, sto=
reD=3D
> >ir, false, suffix, maxCacheKeys,=3D20
> >+ CHKBlock.DATA_LENGTH,
> >CHKBlock.TOTAL_HEADERS_LENGTH, true, Berkele=
y=3D
> >DBFreenetStore.TYPE_CHK, storeEnvironment, random);
> > Logger.normal(this, "Initializing pubKey Datastore");
> > System.out.println("Initializing pubKey Datastore");
> >- pubKeyDatastore =3D3D
> >BerkeleyDBFreenetStore.construct(lastVersion, =
"",=3D
> > storeDir, true, suffix, maxStoreKeys,=3D20
> >- DSAPublicKey.PADDED_SIZE, 0, true,
> >BerkeleyDBFreenetStore.TYPE_PUB=
K=3D
> >EY);
> >+ pubKeyDatastore =3D3D
> >BerkeleyDBFreenetStore.construct(lastVersion, =
sto=3D
> >reDir, true, suffix, maxStoreKeys,=3D20
> >+ DSAPublicKey.PADDED_SIZE, 0, true,
> >BerkeleyDBFreenetStore.TYPE_PUB=
K=3D
> >EY, storeEnvironment, random);
> > Logger.normal(this, "Initializing pubKey Datacache");
> > System.out.println("Initializing pubKey Datacache
> > ("+maxCacheKeys+" =
k=3D
> >eys)");
> >- pubKeyDatacache =3D3D
> >BerkeleyDBFreenetStore.construct(lastVersion, =
"",=3D
> > storeDir, false, suffix, maxCacheKeys,=3D20
> >- DSAPublicKey.PADDED_SIZE, 0, true,
> >BerkeleyDBFreenetStore.TYPE_PUB=
K=3D
> >EY);
> >+ pubKeyDatacache =3D3D
> >BerkeleyDBFreenetStore.construct(lastVersion, =
sto=3D
> >reDir, false, suffix, maxCacheKeys,=3D20
> >+ DSAPublicKey.PADDED_SIZE, 0, true,
> >BerkeleyDBFreenetStore.TYPE_PUB=
K=3D
> >EY, storeEnvironment, random);
> > // FIXME can't auto-fix SSK stores.
> > Logger.normal(this, "Initializing SSK Datastore");
> > System.out.println("Initializing SSK Datastore");
> >- sskDatastore =3D3D
> >BerkeleyDBFreenetStore.construct(lastVersion, "",=
st=3D
> >oreDir, true, suffix, maxStoreKeys,=3D20
> >- SSKBlock.DATA_LENGTH,
> >SSKBlock.TOTAL_HEADERS_LENGTH, false, Berkel=
e=3D
> >yDBFreenetStore.TYPE_SSK);
> >+ sskDatastore =3D3D
> >BerkeleyDBFreenetStore.construct(lastVersion, sto=
reD=3D
> >ir, true, suffix, maxStoreKeys,=3D20
> >+ SSKBlock.DATA_LENGTH,
> >SSKBlock.TOTAL_HEADERS_LENGTH, false, Berkel=
e=3D
> >yDBFreenetStore.TYPE_SSK, storeEnvironment, random);
> > Logger.normal(this, "Initializing SSK Datacache");
> > System.out.println("Initializing SSK Datacache
> > ("+maxCacheKeys+" key=
s=3D
> >)");
> >- sskDatacache =3D3D
> >BerkeleyDBFreenetStore.construct(lastVersion, "",=
st=3D
> >oreDir, true, suffix, maxStoreKeys,=3D20
> >- SSKBlock.DATA_LENGTH,
> >SSKBlock.TOTAL_HEADERS_LENGTH, false, Berkel=
e=3D
> >yDBFreenetStore.TYPE_SSK);
> >+ sskDatacache =3D3D
> >BerkeleyDBFreenetStore.construct(lastVersion, sto=
reD=3D
> >ir, false, suffix, maxStoreKeys,=3D20
> >+ SSKBlock.DATA_LENGTH,
> >SSKBlock.TOTAL_HEADERS_LENGTH, false, Berkel=
e=3D
> >yDBFreenetStore.TYPE_SSK, storeEnvironment, random);
> > } catch (FileNotFoundException e1) {
> > String msg =3D3D "Could not open datastore: "+e1;
> > Logger.error(this, msg, e1);
>=20
> >Modified: trunk/freenet/src/freenet/store/BerkeleyDBFreenetStore.java
> >=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=
=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D
> >=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=
=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D
> >=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=3D3D=
=3D3D=3D3D=3D3D
> >--- trunk/freenet/src/freenet/store/BerkeleyDBFreenetStore.java
> >2006-11-=
1=3D
> >1 17:05:05 UTC (rev 10887)
> >+++ trunk/freenet/src/freenet/store/BerkeleyDBFreenetStore.java
> >2006-11-=
1=3D
> >1 20:46:52 UTC (rev 10888)
> >@@ -29,6 +29,7 @@
> > import com.sleepycat.je.Transaction;
> >=3D20
> > import freenet.crypt.DSAPublicKey;
> >+import freenet.crypt.RandomSource;
> > import freenet.keys.CHKBlock;
> > import freenet.keys.CHKVerifyException;
> > import freenet.keys.KeyBlock;
> >@@ -82,40 +83,229 @@
> > =3D09
> > private boolean closed;
> > private final static byte[] dummy =3D3D new byte[0];
> >+=3D09
> >+ public static BerkeleyDBFreenetStore construct(int lastVersion, File b=
a=3D
> >seStoreDir, boolean isStore,=3D20
> >+ String suffix, long maxStoreKeys, int blockSize, int
> >headerSize, boo=
l=3D
> >ean throwOnTooFewKeys, short type, Environment storeEnvironment, RandomS=
o=3D
> >urce random) throws Exception {
> >=3D20
> >- public static BerkeleyDBFreenetStore construct(int lastVersion, String=
=3D
> >prefix, File baseStoreDir, boolean isStore,=3D20
> >- String suffix, long maxStoreKeys, int blockSize, int
> >headerSize, boo=
l=3D
> >ean throwOnTooFewKeys, short type) throws Exception {
> >+ /**
> >+ * Migration strategy:
> >+ *=3D20
> >+ * If nothing exists, create a new database in the
> >storeEnvironment a=
n=3D
> >d store files of new names.
> >+ * Else
> >+ * If the old store directories exist:
> >+ * If the old store file does not exist, delete the old
> >store di=
r=3D
> >ectories, and create a new database in the storeEnvironment and store fi=
l=3D
> >es of new names.
> >+ * Try to load the old database.
> >+ * If successful
> >+ * Migrate to the new database.
> >+ * Move the files.
> >+ * If not successful
> >+ * Reconstruct the new database from the old file.
> >+ * Move the old file to the new location.
> >+ *=3D20
> >+ */
> >+ =3D09
> >+ // Location of old directory.
> >+ String oldDirName =3D3D oldTypeName(type) + (isStore ? "store"
> >: "cac=
he"=3D
> >) + suffix;
> >+ File oldDir =3D3D new File(baseStoreDir, oldDirName);
> >+ =3D09
> >+ File oldDBDir =3D3D new File(oldDir, "database");
> >+ File oldStoreFile =3D3D new File(oldDir, "store");
> >+ =3D09
> >+ // Location of new store file
> >+ String newStoreFileName =3D3D newTypeName(type) + suffix + "."+
> >(isSt=
ore=3D
> > ? "store" : "cache");
> >+ File newStoreFile =3D3D new File(baseStoreDir,
> >newStoreFileName);
> >=3D20
> >- File dir =3D3D new File(baseStoreDir,=3D20
> >- typeName(type) + (isStore ? "store" : "cache")
> >+ suffix);=3D20
> >+ String newDBPrefix =3D3D newTypeName(type)+"-"+(isStore ?
> >"store" : "=
cac=3D
> >he")+"-";
> > =3D09
> >- if(!dir.exists())
> >- dir.mkdir();
> >+ File newFixSecondaryFile =3D3D new File(baseStoreDir,
> >"recreate_secon=
dar=3D
> >y_db-"+newStoreFileName);
> > =3D09
> >- File dbDir =3D3D new File(dir,"database");
> >- if(!dbDir.exists())
> >- dbDir.mkdir();
> >+ BerkeleyDBFreenetStore tmp;
> > =3D09
> >- Environment env =3D3D null;
> >- // Initialize environment
> >+ if(newStoreFile.exists()) {
> >+ =3D09
> >+ System.err.println("Opening database using
> >"+newStoreFile);
> >+ =3D09
> >+ // Try to load new database, reconstruct it if
> >necessary.
> >+ // Don't need to create a new Environment, since we can
> >use the old =
o=3D
> >ne.
> >+ =3D09
> >+ tmp =3D3D openStore(storeEnvironment, newDBPrefix,
> >newStoreFile, new=
Fix=3D
> >SecondaryFile, maxStoreKeys,
> >+ blockSize, headerSize,
> >throwOnTooFewKeys, false, lastVersion, type=
,=3D
> > false);
> >+ =3D09
> >+ } else if(oldDir.exists() && oldStoreFile.exists()) {
> >+ =3D09
> >+ System.err.println("Old directory exists");
> >+ =3D09
> >+ File storeFile =3D3D newStoreFile;
> >+ =3D09
> >+ // Move old store file to new location.
> >+ =3D09
> >+ if(!oldStoreFile.renameTo(newStoreFile)) {
> >+ System.err.println("Cannot move store file from
> >"+oldStoreFile+" to=
=3D
> >"+newStoreFile);
> >+ // Use old location for now.
> >+ storeFile =3D3D oldStoreFile;
> >+ // Will block deletion below.
> >+ } else {
> >+ System.err.println("Moved store file from
> >"+oldStoreFile+" to "+new=
S=3D
> >toreFile);
> >+ }
> >+ =3D09
> >+ if(oldDBDir.exists()) {
> >+ =3D09
> >+ // Try to open old database with new store file.
> >+ // If database is invalid, do below.
> >+ // Otherwise, copy data from old database to
> >new database.
> >+ =3D09
> >+ // Open the old store
> >+ =3D09
> >+ Environment oldEnv =3D3D null;
> >+ // Initialize environment
> >+ try {
> >+ EnvironmentConfig envConfig =3D3D new
> >EnvironmentConfig();
> >+ envConfig.setAllowCreate(true);
> >+ envConfig.setTransactional(true);
> >+ envConfig.setTxnWriteNoSync(true);
> >+ oldEnv =3D3D new Environment(oldDBDir,
> >envConfig);
> >+ } catch (DatabaseException e) {
> >+ if(oldEnv !=3D3D null)
> >+ oldEnv.close();
> >+ throw e;
> >+ }
> >+ =3D09
> >+ // Initialize CHK database
> >+ DatabaseConfig dbConfig =3D3D new
> >DatabaseConfig();
> >+ dbConfig.setAllowCreate(true);
> >+ dbConfig.setTransactional(true);
> >+ Database oldChkDB =3D3D
> >oldEnv.openDatabase(null,"CHK",dbConfig);
> >+ =3D09
> >+ // Open the new store
> >+ tmp =3D3D openStore(storeEnvironment,
> >newDBPrefix, storeFile, newFi=
xSe=3D
> >condaryFile, maxStoreKeys,
> >+ blockSize, headerSize,
> >false, true, lastVersion, type, true);
> >+ =3D09
> >+ // Migrate all tuples from old database to new
> >database.
> >+ migrateTuples(oldEnv, oldChkDB, tmp);
> >+ =3D09
> >+ oldChkDB.close();
> >+ =3D09
> >+ oldEnv.close();
> >+
> >+ tmp.checkForHoles(tmp.countCHKBlocksFromFile());
> >+ tmp.maybeShrink(true, true);
> >+ =3D09
> >+ } else {
> >+ =3D09
> >+ // No old database to worry about.
> >+ // Reconstruct the new database from the store
> >file which is now in=
=3D
> >the right place.
> >+ =3D09
> >+ tmp =3D3D openStore(storeEnvironment,
> >newDBPrefix, storeFile, newFi=
xSe=3D
> >condaryFile, maxStoreKeys,
> >+ blockSize, headerSize, true,
> >false, lastVersion, type, false);
> >+ =3D09
> >+ }
> >+ =3D09
> >+ } else {
> >+ =3D09
> >+ // No new store file, no new database.
> >+ // Start from scratch, with new store.
> >+ =3D09
> >+ tmp =3D3D openStore(storeEnvironment, newDBPrefix,
> >newStoreFile, new=
Fix=3D
> >SecondaryFile, maxStoreKeys,
> >+ blockSize, headerSize, true, false,
> >lastVersion, type, false);
> >+ =3D09
> >+ }
> >+
> >+ // Delete old store directory
> >+ deleteOldStoreDir(baseStoreDir, oldDBDir, oldDir, oldDirName,
> >random);
> >+ =3D09
> >+ return tmp;
> >+ }
> >+
> >+ private static void migrateTuples(Environment oldEnv, Database oldChkD=
B=3D
> >, BerkeleyDBFreenetStore newStore) throws DatabaseException {
> >+
> >+ System.err.println("Migrating data from old Environment to new
> >Enviro=
n=3D
> >ment");
> >+ /** Reads from old database */
> >+ Cursor c =3D3D null;
> >+ /** Writes to new store */
> >+ Transaction t =3D3D null;
> > try {
> >- EnvironmentConfig envConfig =3D3D new
> >EnvironmentConfig();
> >- envConfig.setAllowCreate(true);
> >- envConfig.setTransactional(true);
> >- envConfig.setTxnWriteNoSync(true);
> >- env =3D3D new Environment(dbDir, envConfig);
> >+ // Read from old database
> >+ t =3D3D newStore.environment.beginTransaction(null,
> >null);
> >+ //t =3D3D oldEnv.beginTransaction(null,null);
> >+ c =3D3D oldChkDB.openCursor(null,null);
> >+ DatabaseEntry keyDBE =3D3D new DatabaseEntry();
> >+ DatabaseEntry blockDBE =3D3D new DatabaseEntry();
> >+ OperationStatus opStat;
> >+ opStat =3D3D c.getFirst(keyDBE, blockDBE,
> >LockMode.DEFAULT);
> >+ if(opStat =3D3D=3D3D OperationStatus.NOTFOUND) {
> >+ System.err.println("Database is empty
> >(migrating tuples).");
> >+ c.close();
> >+ c =3D3D null;
> >+ return;
> >+ }
> >+ if(logMINOR) Logger.minor(BerkeleyDBFreenetStore.class,
> >"Found first=
=3D
> >key");
> >+ int x =3D3D 0;
> >+ while(true) {
> >+ opStat =3D3D newStore.chkDB.putNoOverwrite(t,
> >keyDBE, blockDBE);
> >+ if(opStat =3D3D=3D3D OperationStatus.KEYEXIST) {
> >+ System.err.println("Duplicate key");
> >+ } else if(opStat =3D3D=3D3D
> >OperationStatus.KEYEMPTY) {
> >+ System.err.println("Key empty");
> >+ } else if(opStat =3D3D=3D3D
> >OperationStatus.NOTFOUND) {
> >+ System.err.println("Not found");
> >+ } else if(opStat =3D3D=3D3D
> >OperationStatus.SUCCESS) {
> >+ // It worked, cool.
> >+ } else {
> >+ throw new Error("Unknown
> >OperationStatus: "+opStat);
> >+ }
> >+ opStat =3D3D c.getNext(keyDBE, blockDBE,
> >LockMode.RMW);
> >+ x++;
> >+ if(x % 512 =3D3D=3D3D 0) {
> >+ t.commit();
> >+ t =3D3D
> >newStore.environment.beginTransaction(null, null);
> >+ }
> >+ if(opStat =3D3D=3D3D OperationStatus.NOTFOUND) {
> >+ System.err.println("Completed
> >migration.");
> >+ return;
> >+ }
> >+ }
> > } catch (DatabaseException e) {
> >- if(env !=3D3D null)
> >- env.close();
> >+ System.err.println("Caught: "+e);
> >+ e.printStackTrace();
> >+ Logger.error(BerkeleyDBFreenetStore.class, "Caught "+e,
> >e);
> >+ try {
> >+ t.abort();
> >+ } catch (DatabaseException e1) {
> >+ System.err.println("Failed to abort: "+e1);
> >+ e1.printStackTrace();
> >+ }
> >+ t =3D3D null;
> > throw e;
> >+ } finally {
> >+ if(c !=3D3D null) {
> >+ try {
> >+ c.close();
> >+ } catch (DatabaseException e) {
> >+ System.err.println("Cannot close
> >cursor: "+e);
> >+ e.printStackTrace();
> >+ }
> >+ }
> >+ if(t !=3D3D null) {
> >+ try {
> >+ t.commit();
> >+ } catch (DatabaseException e) {
> >+ System.err.println("Cannot close
> >transaction: "+t);
> >+ e.printStackTrace();
> >+ }
> >+ }
> > }
> >+ =3D09
> >+ }
> >=3D20
> >- BerkeleyDBFreenetStore tmp;
> >+ private static BerkeleyDBFreenetStore openStore(Environment storeEnvir=
o=3D
> >nment, String newDBPrefix, File newStoreFile,=3D20
> >+ File newFixSecondaryFile, long maxStoreKeys, int
> >blockSize, int head=
e=3D
> >rSize, boolean throwOnTooFewKeys, boolean noCheck, int lastVersion, shor=
t=3D
> > type, boolean wipe) throws Exception {
> >+ =3D09
> > try {
> > if((lastVersion > 0) && (lastVersion < 852)) {
> > throw new DatabaseException("Reconstructing
> > store because started f=
r=3D
> >om old version");
> > }
> >- tmp =3D3D new BerkeleyDBFreenetStore(env, prefix, dir,
> >maxStoreKeys,=
bl=3D
> >ockSize, headerSize, throwOnTooFewKeys);
> >+ return new BerkeleyDBFreenetStore(storeEnvironment,
> >newDBPrefix, new=
S=3D
> >toreFile, newFixSecondaryFile,=3D20
> >+ maxStoreKeys, blockSize, headerSize,
> >throwOnTooFewKeys, noCheck, w=
i=3D
> >pe);
> > } catch (DatabaseException e) {
> > =3D09
> > System.err.println("Could not open store: "+e);
> >@@ -131,41 +321,56 @@
> > =3D09
> > // Reconstruct
> > =3D09
> >- // First, close it.
> >- try {=3D20
> >- env.close();
> >- } catch (Throwable t) {
> >- // Ignore, probably double-closing
> >- // FIXME shouldn't be necessary
> >- }
> >- =3D09
> >- // Now delete the old database
> >+ return new BerkeleyDBFreenetStore(storeEnvironment,
> >newDBPrefix, new=
S=3D
> >toreFile, newFixSecondaryFile, maxStoreKeys, blockSize, headerSize, type=
,=3D
> > noCheck);
> >+ }
> >+ }
> >=3D20
> >- if(dbDir.exists()) {
> >- File[] files =3D3D dbDir.listFiles();
> >- for(int i=3D3D0;i<files.length;i++) {
> >- if(!files[i].delete())
> >- System.err.println("Failed to
> >delete "+files[i]);
> >+ private static void deleteOldStoreDir(File baseStoreDir, File oldDBDir=
,=3D
> > File oldDir, String oldDirName, RandomSource random) {
> >+ if(!oldDir.exists()) return;
> >+ System.err.println("Deleting old store dir: "+oldDir);
> >+ // Delete
> >+ boolean deleteFailed =3D3D false;
> >+ if(oldDBDir.exists()) {
> >+ File[] list =3D3D oldDBDir.listFiles();
> >+ for(int i=3D3D0;i<list.length;i++) {
> >+ File f =3D3D list[i];
> >+ String name =3D3D f.getName();
> >+ if(name.equals("je.lck") ||
> >name.endsWith(".jdb")) {
> >+ if(!f.delete()) {
> >+ if(f.exists()) {
> >+
> >System.err.println("Failed to delete old database file "+f+" (no=
=3D
> >store file so old database worthless)");
> >+ deleteFailed =3D3D true;
> >+ }
> >+ }
> >+ } else {
> >+ System.err.println("Did not delete
> >unknown file "+f+" - created by=
=3D
> >user?");
> >+ deleteFailed =3D3D true;
> > }
> >- } else
> >- dbDir.mkdir();
> >- =3D09
> >- // Now create a new one.
> >- =3D09
> >- // Initialize environment
> >- EnvironmentConfig envConfig =3D3D new
> >EnvironmentConfig();
> >- envConfig.setAllowCreate(true);
> >- envConfig.setTransactional(true);
> >- envConfig.setTxnWriteNoSync(true);
> >-
> >- env =3D3D new Environment(dbDir, envConfig);
> >- =3D09
> >- tmp =3D3D new BerkeleyDBFreenetStore(env, dir, dbDir,
> >maxStoreKeys, =
blo=3D
> >ckSize, headerSize, type);
> >+ }
> >+ if(!deleteFailed) {
> >+ if(!oldDBDir.delete()) {
> >+ System.err.println("Unable to delete
> >database directory: "+oldDBDi=
r=3D
> >+" (no store file so old database worthless)");
> >+ deleteFailed =3D3D true;
> >+ }
> >+ }
> > }
> >- return tmp;
> >+ if(deleteFailed) {
> >+ // Try to rename the old directory
> >+ File f =3D3D new File(baseStoreDir,
> >"lost+found-"+oldDirName);
> >+ while(f.exists()) {
> >+ f =3D3D new File(baseStoreDir,
> >"lost+found-"+oldDirName+"-"+Long.to=
Hex=3D
> >String(random.nextLong()));
> >+ }
> >+ if(!oldDir.renameTo(f)) {
> >+ System.err.println("Unable to rename old store
> >directory "+oldDir+"=
=3D
> >to "+f+" (would have deleted it but it has user files or is not deletabl=
e=3D
> >)");
> >+ }
> >+ } else {
> >+ if(!oldDir.delete()) {
> >+ System.err.println("Unable to delete old store
> >directory "+oldDir+"=
=3D
> >(no useful data)");
> >+ }
> >+ }
> > }
> >=3D20
> >- private static String typeName(short type) {
> >+ private static String oldTypeName(short type) {
> > if(type =3D3D=3D3D TYPE_CHK)
> > return "";
> > else if(type =3D3D=3D3D TYPE_SSK)
> >@@ -175,41 +380,26 @@
> > else throw new Error("No such type "+type);
> > }
> >=3D20
> >- public static BerkeleyDBFreenetStore construct(String prefix, String s=
t=3D
> >oreDir, long maxChkBlocks, int blockSize, int headerSize, boolean throwO=
n=3D
> >TooFewKeys) throws IOException, DatabaseException {
> >-
> >- File dir =3D3D new File(storeDir);
> >- if(!dir.exists())
> >- dir.mkdir();
> >- =3D09
> >- File dbDir =3D3D new File(dir,"database");
> >- if(!dbDir.exists())
> >- dbDir.mkdir();
> >- =3D09
> >- Environment env =3D3D null;
> >- // Initialize environment
> >- try {
> >- EnvironmentConfig envConfig =3D3D new
> >EnvironmentConfig();
> >- envConfig.setAllowCreate(true);
> >- envConfig.setTransactional(true);
> >- envConfig.setTxnWriteNoSync(true);
> >- env =3D3D new Environment(dbDir, envConfig);
> >- } catch (DatabaseException e) {
> >- if(env !=3D3D null)
> >- env.close();
> >- throw e;
> >- }
> >-
> >- return new BerkeleyDBFreenetStore(env, prefix, dir,
> >maxChkBlocks, blo=
c=3D
> >kSize, headerSize, throwOnTooFewKeys);
> >+ private static String newTypeName(short type) {
> >+ if(type =3D3D=3D3D TYPE_CHK)
> >+ return "chk";
> >+ else if(type =3D3D=3D3D TYPE_SSK)
> >+ return "ssk";
> >+ else if(type =3D3D=3D3D TYPE_PUBKEY)
> >+ return "pubkey";
> >+ else throw new Error("No such type "+type);
> > }
> > =3D09
> > /**
> > * Initializes database
> >+ * @param noCheck If true, don't check for holes etc.
> >+ * @param wipe If true, wipe the database first.
> > * @param the directory where the store is located
> > * @throws IOException=3D20
> > * @throws DatabaseException=3D20
> > * @throws FileNotFoundException if the dir does not exist and coul=
d=3D
> > not be created
> > */
> >- public BerkeleyDBFreenetStore(Environment env, String prefix, File sto=
r=3D
> >eDir, long maxChkBlocks, int blockSize, int headerSize, boolean throwOnT=
o=3D
> >oFewKeys) throws IOException, DatabaseException {
> >+ public BerkeleyDBFreenetStore(Environment env, String prefix, File sto=
r=3D
> >eFile, File fixSecondaryFile, long maxChkBlocks, int blockSize, int head=
e=3D
> >rSize, boolean throwOnTooFewKeys, boolean noCheck, boolean wipe) throws =
I=3D
> >OException, DatabaseException {
> > logMINOR =3D3D Logger.shouldLog(Logger.MINOR, this);
> > this.dataBlockSize =3D3D blockSize;
> > this.headerBlockSize =3D3D headerSize;
> >@@ -223,15 +413,31 @@
> > DatabaseConfig dbConfig =3D3D new DatabaseConfig();
> > dbConfig.setAllowCreate(true);
> > dbConfig.setTransactional(true);
> >+ if(wipe) {
> >+ try {
> >+ environment.removeDatabase(null,prefix+"CHK");
> >+ } catch (DatabaseException e) {
> >+ Logger.error(this, "Could not remove
> >"+prefix+"CHK", e);
> >+ }
> >+ try {
> >+
> >environment.removeDatabase(null,prefix+"CHK_accessTime");
> >+ } catch (DatabaseException e) {
> >+ Logger.error(this, "Could not remove
> >"+prefix+"CHK_accessTime", e);
> >+ }
> >+ try {
> >+
> >environment.removeDatabase(null,prefix+"CHK_blockNum");
> >+ } catch (DatabaseException e) {
> >+ Logger.error(this, "Could not remove
> >"+prefix+"CHK_blockNum", e);
> >+ }
> >+ }
> > chkDB =3D3D
> > environment.openDatabase(null,prefix+"CHK",dbConfig);
> >- =3D09
> >- fixSecondaryFile =3D3D new File(storeDir,
> >"recreate_secondary_db");
> >- =3D09
> >+
> >+ this.fixSecondaryFile =3D3D fixSecondaryFile;
> > if(fixSecondaryFile.exists()) {
> > fixSecondaryFile.delete();
> >- Logger.error(this, "Recreating secondary database for
> >"+storeDir);
> >+ Logger.error(this, "Recreating secondary database");
> > Logger.error(this, "This may take some time...");
> >- System.err.println("Recreating secondary database for
> >"+storeDir);
> >+ System.err.println("Recreating secondary database");
> > System.err.println("This may take some time...");
> > try {
> > environment.truncateDatabase(null,
> > prefix+"CHK_accessTime", false);
> >@@ -281,7 +487,7 @@
> > } catch (DatabaseNotFoundException e) {
> > System.err.println("Migrating block db index");
> > // De-dupe on keys and block numbers.
> >- migrate(storeDir);
> >+ migrate();
> > System.err.println("De-duped, creating new index...");
> > blockNoDbConfig.setSortedDuplicates(false);
> > blockNoDbConfig.setAllowCreate(true);
> >@@ -296,7 +502,6 @@
> > chkDB_blockNum =3D3D blockNums;
> > =3D09
> > // Initialize the store file
> >- File storeFile =3D3D new File(storeDir,"store");
> > try {
> > if(!storeFile.exists())
> > storeFile.createNewFile();
> >@@ -318,7 +523,7 @@
> > t.printStackTrace();
> > }
> > throw new DatabaseException("Keys in
> > database: "+chkBlocksInStore+=
"=3D
> > but keys in file: "+chkBlocksFromFile);
> >- } else {
> >+ } else if(!noCheck) {
> > long len =3D3D
> > checkForHoles(chkBlocksFromFile);
> > if(len < chkBlocksFromFile) {
> > System.err.println("Truncating
> > to "+len+" as no non-holes after t=
h=3D
> >at point");
> >@@ -330,10 +535,14 @@
> > =3D09
> > chkBlocksInStore =3D3D Math.max(chkBlocksInStore,
> > chkBlocksFromFile);
> > if(logMINOR) Logger.minor(this, "Keys in store:
> > "+chkBlocksInStore);
> >- System.out.println("Keys in store: "+chkBlocksInStore+"
> >/ "+maxChkBl=
o=3D
> >cks+" (db "+chkBlocksInDatabase+" file "+chkBlocksFromFile+")");
> >-
> >- maybeShrink(true, true);
> >+ System.out.println("Keys in store: db
> >"+chkBlocksInDatabase+" file "=
+=3D
> >chkBlocksFromFile+" / max "+maxChkBlocks);
> > =3D09
> >+ if(!noCheck) {
> >+ maybeShrink(true, true);
> >+ chkBlocksFromFile =3D3D
> >countCHKBlocksFromFile();
> >+ chkBlocksInStore =3D3D
> >Math.max(chkBlocksInStore, chkBlocksFromFile=
);
> >+ }
> >+ =3D09
> > // Add shutdownhook
> > Runtime.getRuntime().addShutdownHook(new
> > ShutdownHook());
> > } catch (DatabaseException t) {
> >@@ -413,7 +622,7 @@
> > opStat =3D3D c.getLast(keyDBE, blockDBE, LockMode.RMW);
> > =3D09
> > if(opStat =3D3D=3D3D OperationStatus.NOTFOUND) {
> >- System.err.println("Database is empty.");
> >+ System.err.println("Database is empty
> >(shrinking).");
> > c.close();
> > c =3D3D null;
> > return;
> >@@ -597,12 +806,15 @@
> > else
> > t =3D3D null;
> > }
> >+
> >+ freeBlocks.remove(i);
> > =3D09
> > synchronized(this) {
> > maxBlocks =3D3D maxChkBlocks;
> > curBlocks =3D3D
> > chkBlocksInStore;
> > if(maxBlocks >=3D3D curBlocks)
> > break;
> > }
> >+ =3D09
> > }
> > =3D09
> > t.commit();
> >@@ -642,7 +854,7 @@
> > * @param the directory where the store is located
> > * @throws FileNotFoundException if the dir does not exist and coul=
d=3D
> > not be created
> > */
> >- public BerkeleyDBFreenetStore(Environment env, File dir, File dbDir, l=
o=3D
> >ng maxChkBlocks, int blockSize, int headerSize, short type) throws Excep=
t=3D
> >ion {
> >+ public BerkeleyDBFreenetStore(Environment env, String prefix, File sto=
r=3D
> >eFile, File fixSecondaryFile, long maxChkBlocks, int blockSize, int head=
e=3D
> >rSize, short type, boolean noCheck) throws Exception {
> > logMINOR =3D3D Logger.shouldLog(Logger.MINOR, this);
> > this.dataBlockSize =3D3D blockSize;
> > this.headerBlockSize =3D3D headerSize;
> >@@ -654,9 +866,9 @@
> > DatabaseConfig dbConfig =3D3D new DatabaseConfig();
> > dbConfig.setAllowCreate(true);
> > dbConfig.setTransactional(true);
> >- chkDB =3D3D environment.openDatabase(null,"CHK",dbConfig);
> >+ chkDB =3D3D
> >environment.openDatabase(null,prefix+"CHK",dbConfig);
> > =3D09
> >- fixSecondaryFile =3D3D new File(dir, "recreate_secondary_db");
> >+ this.fixSecondaryFile =3D3D fixSecondaryFile;
> > fixSecondaryFile.delete();
> > =3D09
> > // Initialize secondary CHK database sorted on accesstime
> >@@ -671,11 +883,11 @@
> > new AccessTimeKeyCreator(storeBlockTupleBinding);
> > secDbConfig.setKeyCreator(accessTimeKeyCreator);
> > chkDB_accessTime =3D3D environment.openSecondaryDatabase
> >- (null,
> >"CHK_accessTime", chkDB, secDbConfig);
> >+ (null,
> >prefix+"CHK_accessTime", chkDB, secDbConfig);
> > =3D09
> > // Initialize other secondary database sorted on block number
> > try {
> >- environment.removeDatabase(null, "CHK_blockNum");
> >+ environment.removeDatabase(null, prefix+"CHK_blockNum");
> > } catch (DatabaseNotFoundException e) { };
> > SecondaryConfig blockNoDbConfig =3D3D new SecondaryConfig();
> > blockNoDbConfig.setAllowCreate(true);
> >@@ -688,10 +900,9 @@
> > blockNoDbConfig.setKeyCreator(bnkc);
> > System.err.println("Creating block db index");
> > chkDB_blockNum =3D3D environment.openSecondaryDatabase
> >- (null, "CHK_blockNum", chkDB, blockNoDbConfig);
> >+ (null, prefix+"CHK_blockNum", chkDB, blockNoDbConfig);
> > =3D09
> > // Initialize the store file
> >- File storeFile =3D3D new File(dir,"store");
> > if(!storeFile.exists())
> > storeFile.createNewFile();
> > chkStore =3D3D new RandomAccessFile(storeFile,"rw");
> >@@ -700,18 +911,26 @@
> > =3D09
> > lastRecentlyUsed =3D3D 0;
> > =3D09
> >- reconstruct(type, dir);
> >+ reconstruct(type);
> > =3D09
> > chkBlocksInStore =3D3D countCHKBlocksFromFile();
> > lastRecentlyUsed =3D3D getMaxRecentlyUsed();
> > =3D09
> >- maybeShrink(true, true);
> >+ if(!noCheck) {
> >+ long len =3D3D checkForHoles(chkBlocksInStore);
> >+ if(len < chkBlocksInStore) {
> >+ System.err.println("Truncating to "+len+" as no
> >non-holes after tha=
t=3D
> > point");
> >+ chkStore.setLength(len * (dataBlockSize +
> >headerBlockSize));
> >+ chkBlocksInStore =3D3D len;
> >+ }
> >+ maybeShrink(true, true);
> >+ }
> > =3D09
> > // Add shutdownhook
> > Runtime.getRuntime().addShutdownHook(new ShutdownHook());
> > }
> > =3D09
> >- private void reconstruct(short type, File storeDir) throws DatabaseExc=
e=3D
> >ption {
> >+ private void reconstruct(short type) throws DatabaseException {
> > if(type =3D3D=3D3D TYPE_SSK) {
> > System.err.println("Reconstruction of SSK store not
> > supported at pre=
s=3D
> >ent.");
> > throw new UnsupportedOperationException("Reconstruction
> > of SSK store=
=3D
> >not supported at present.");
> >@@ -768,7 +987,7 @@
> > }
> > }
> > } catch (EOFException e) {
> >- migrate(storeDir);
> >+ migrate();
> > return;
> > } catch (IOException e) {
> > Logger.error(this, "Caught "+e, e);
> >@@ -784,9 +1003,9 @@
> > *=3D20
> > * FIXME: Create a list of reusable block numbers?
> > */
> >- private void migrate(File storeDir) throws DatabaseException {
> >+ private void migrate() throws DatabaseException {
> > =3D09
> >- System.err.println("Migrating database "+storeDir+": Creating
> >unique =
i=3D
> >ndex on block number");
> >+ System.err.println("Migrating database: Creating unique index
> >on bloc=
k=3D
> > number");
> > HashSet s =3D3D new HashSet();
> > =3D09
> > Cursor c =3D3D null;
> >@@ -799,7 +1018,7 @@
> > OperationStatus opStat;
> > opStat =3D3D c.getLast(keyDBE, blockDBE, LockMode.RMW);
> > if(opStat =3D3D=3D3D OperationStatus.NOTFOUND) {
> >- System.err.println("Database is empty.");
> >+ System.err.println("Database is empty
> >(migrating).");
> > c.close();
> > c =3D3D null;
> > t.abort();
>=20
> >_______________________________________________
> >cvs mailing list
> >cvs at freenetproject.org
> >http://emu.freenetproject.org/cgi-bin/mailman/listinfo/cvs
>=20
>=20
>=20
>=20
--6c2NcOVqGQ03X4Wi
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: Digital signature
Content-Disposition: inline
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (GNU/Linux)
iD8DBQFFVj6DA9rUluQ9pFARAm5PAKC9AF1tOaXyfyMZzoInF1njnRUBHwCcCLIo
ovhSPMQx94qAmV0Tx4pWmDc=
=vUaT
-----END PGP SIGNATURE-----
--6c2NcOVqGQ03X4Wi--
----- End forwarded message -----
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: Digital signature
URL:
<https://emu.freenetproject.org/pipermail/devl/attachments/20061111/eb58e2a5/attachment.pgp>
