Author: toad
Date: 2008-09-01 17:48:27 +0000 (Mon, 01 Sep 2008)
New Revision: 22306
Modified:
trunk/freenet/src/freenet/node/Version.java
Log:
1159: (Mandatory Monday 8 September 0:00 UTC)
Temp buckets:
- Keeping some temp buckets in RAM automatically was introduced in approx 1158
by nextgens.
- Increase the default max size of a temp bucket kept in RAM to 128K.
- Significant refactoring.
- Get rid of size limit on ArrayBucket's.
- Check for already closed in ArrayBucket's.
- TempBucketFactory.TempBucket now has its own streams.
- Buckets are automatically and transparently migrated when they get too big.
- Buckets are also migrated when they are over 5 minutes old, when creating new
buckets (but off-thread).
- Even if temp bucket encryption is on, temp buckets in RAM are not encrypted
until they are migrated.
- Should fix OOMs people have been reporting with 1158.
- Fix synchronization on ArrayBucket's.
- Track input streams and close them when migrating (inside the lock).
Client layer:
- USKFetcher.removeSubscriber() doesn't need to cancel(), caller will do that,
fixes onCancelled() errors.
- USK callbacks: catch exception, log it, move on to the next callback.
- Switch max archive size back to 2MB. 5MB is too much to wait for for a
freesite to load.
- Fix an NPE in FreenetURI.
- Minor fixes in SimpleManifestPutter.PutHandler.start().
Updater:
- Allow 3 hours (up from 1) before activating UOM (if non-mandatory).
- Fetch freenet-ext #21
Optimisations:
- Introduce a separate lock for the heavily contended
Announcer.timeGotEnoughPeers variable. This seems to help.
- Use StringBuilder not StringBuffer in some places. (Trivial)
- Reduce mark length in LineReadingInputStram. Setting it to integer.max_value
might have caused OOMs with FCP in some cases.
- Don't try to strip the DSA group if it's not there (it's only there if
heavySetup=true).
- Don't write the peers file more than once every 5 seconds.
Content filter:
- Use readFully() not read() in image filters. This might have caused problems
for some images for some bucket implementations.
- Use ArrayList not Vector some places.
L10n:
- Partial Finnish translation from Smar.
- Minor German update from NEOatNHNG.
- French update from batosai.
- Italin update from Luke771.
Stats:
- Ignore thread number when grouping threads for counts.
- Add @Override's.
Plugins:
- New interfaces FredPluginL10n and FredPluginThemed, for L10n integration and
theming of stuff that looks like part of the web interface (but doesn't want to
use PageMaker).
- Plugin mode for PageMaker for plugins that want to do it the right way.
- Forgot to close a stream writing a file loading the CA for SSL downloads.
Dev stuff:
- Bug #2516: Add probablyInStore() to FreenetStore API. Will be used by db4o
branch and maybe other code to avoid heavy lookups when not needed.
- Logging.
- Imports.
- Minor refactorings/code simplifications/dead code removal/variable renaming.
- Use Closer. In one or two cases this may have a functional impact as if one
throws the others won't be closed.
- @Override's.
- Paranoia/sanity checks.
Dev stuff: 1.5isms:
- Use generics in various places.
- Use an enum (with methods and properties) for L10n language selection.
- Make fproxy theme an enum.
Compatibility/SSL:
- Use reflection to access the sun methods we need for cert generation. This
will fail cleanly on non-sun JVMs, and hopefully will make SSL work on Sun ones.
XMLSpider:
- Version 32: Increase the maximum sub-index size to 4MB before compression.
This will likely compress to <<500K. This should make the indexes use fewer
files, avoid problems with too many files (in the current code), and improve
efficiency.
UP&P:
- Use some generics.
- Add a main() method for debugging.
- Don't return bandwidth limits if double-NATed.
KeyExplorer, ThawIndexBrowser:
- Use PageMaker properly, using the callback.
- Implement FredPluginL10n.
JSTUN:
- Add version string.
Installer:
- Show a new google-hosted uninstall survey on removal on both Windows and *nix.
- Move outdated windows installer executables out of the way (from
trunk/win32-bin to legacy).
nextgens
saces
toad
j16sdiz
Smar
NEOatNHNG
Luke771
batosai
Other recent SVN activity since 1158:
Web of Trust plugin (batosai)
- Various misc fixes.
- Version 0.1.2.
- Stop updating ownIdentities score in their own trustList. Might help with bug
#2517.
- More testing code.
- Page to show/edit details for an OwnIdentity.
- Store only the unique part of the request URI in the database.
- Ability to restore an identity from Freenet given request/insert URIs.
- Logging.
Salted hash datastore branch (j16sdiz)
- Various misc fixes.
- probablyInStore().
- Merges.
- Config options for size and counting.
Db4o branch (toad)
- Minor optimisations.
- Logging.
- Use (Java 1.5) generics.
- MAJOR optimisation: Use Bloom filters for splitfiles. This costs
approximately an extra 3MB permanent RAM per queued 10GB of downloads. It
greatly improves performance by avoiding a vast number of queries and object
creations. We no longer have a long period of high CPU, high object churn and
high disk I/O after adding a big splitfile to the queue, and we have much
reduced background disk I/O and object churn.
-- KeyListener: a transient object kept in RAM that identifies whether a key is
of interest to a specific high-level request. probablyWantKey() is expected to
be a fast, non-database, non-I/O, minimal locking lookup. We call this on all
the KeyListener's, and if any want it, we schedule a job on the database thread
for definitelyWantKey() (for ULPRs) or getRequestsForKey() (for retrying after
cooldown) or handleBlock() (when we've found the data). HasKeyListener
indicates that a KeyListener can be constructed; we create one for all
persistent requests on startup.
-- Note that this scales O(n) with the number of parallel download requests
(high level requests e.g. a 4GB splitfile counts as 1). The number of downloads
is unlikely to exceed a few thousand, and for any practical workload and
especially with the current common case of fast CPUs and slow disks, this is
vastly faster than using an on-disk index, as demonstrated above.
-- There are possibilities for improving on this further in the future, such as
maintaining an overall Bloom filter, but they would involve more work.
-- Splitfiles use one (counting, 19 elements per key i.e 0.01% false positives)
Bloom filter to identify which keys belong to it, plus one non-counting filter
per segment (size determined to ensure a 1% false positives for all segments
combined). We can move the per-segment filters to disk if need be, but we would
need to be careful to read all of them at once to avoid seeking. Note that the
bloom filters for persistent requests are kept on disk, and are not encrypted
(but are salted).
- The datastore checker is now a first class object. We have one and only one,
which has a single thread. Transient requests are queued; persistent requests
get a DatastoreCheckerItem stored, a limited number of keys are stored in RAM,
and more can be pulled from disk by a query, so the total amount of RAM used by
the queue is severely limited. Once a getter's keys have been checked in the
store, it is passed on to finish registration in CRS. Getters' KeyListeners
will have been registered before checking the store. We still keep RegisterMe
but we only use it for inserts. Retries pass a flag to not check the store.
- Lots of minor optimisations, bug fixes, memory leak fixes etc.
- Major counting bloom filter fix, arithmetic problem causing only half the
bits to be used and many many false positives. Code was from the salted hash
store branch.
- Fix a problem with the node trying to decompress a file twice because the
metadata was a compressed splitfile and it kept the extra decompressor on the
list.
Modified: trunk/freenet/src/freenet/node/Version.java
===================================================================
--- trunk/freenet/src/freenet/node/Version.java 2008-09-01 16:07:56 UTC (rev
22305)
+++ trunk/freenet/src/freenet/node/Version.java 2008-09-01 17:48:27 UTC (rev
22306)
@@ -24,17 +24,17 @@
public static final String protocolVersion = "1.0";
/** The build number of the current revision */
- private static final int buildNumber = 1158;
+ private static final int buildNumber = 1159;
/** Oldest build of Fred we will talk to */
- private static final int oldLastGoodBuild = 1155;
- private static final int newLastGoodBuild = 1158;
+ private static final int oldLastGoodBuild = 1158;
+ private static final int newLastGoodBuild = 1159;
static final long transitionTime;
static {
final Calendar _cal =
Calendar.getInstance(TimeZone.getTimeZone("GMT"));
// year, month - 1 (or constant), day, hour, minute, second
- _cal.set( 2008, Calendar.AUGUST, 20, 0, 0, 0 );
+ _cal.set( 2008, Calendar.SEPTEMBER, 8, 0, 0, 0 );
transitionTime = _cal.getTimeInMillis();
}
