Author: toad Date: 2009-01-10 22:29:24 +0000 (Sat, 10 Jan 2009) New Revision: 25017
Modified: trunk/freenet/src/freenet/node/Version.java Log: 1195: (MANDATORY MONDAY 0:00 GMT) Request latency: - Track the average time taken for a successful vs unsuccessful vs average local CHK request. - Current successful average is around 18 seconds on my node! - Transfer speed optimiser: Target 5 seconds, don't accept more requests than can transfer all their data in 5 seconds, that target is increased for really slow nodes (not needed for 12K/sec+ though). - Objective is to improve the time taken to transfer a block. Until the build is widely deployed the network will be slowed down significantly, but after that we should have faster requests, faster latency, faster fproxy and comparable to present output bandwidth - we hope! We may have to revert this, but it's an easy experiment that might greatly improve fproxy performance... History cloaking: - Because we have got rid of the firefox profile, and because many people will access Freenet without using the firefox profile, we have to worry about the possibility of browser history stealing. - The solution is to add a ?secureid= parameter to each page. This is the Base64-encoded first 12 bytes of the SHA256 of a node-specific private nonce (stored in client-nonce.dat) and the URI being fetched (with some re-encoding to be consistent despite different standards, browsers etc). - If the user does not specify this, we tell them, asking them to clear their browser history, and explaining that ideally they would use separate browsers and could then turn this mechanism off. Message depends on whether it's no secureid or a broken one. - Write the url of / to freenet.url.dat. This will be used by browse.cmd / browse.sh to open the correct start page, once the build is deployed. - Some entropy issues with generating the nonce, since we need to start the web interface before seeding Yarrow; if we have urandom, use that, use SecureRandom.generateKey if necessary (blocking on non-windows); start the entropy gathering thread early if needed. - LinkFixer interface, must be used when sending a link to the browser. - History cloaking is configurable and off by default. Installers will set it by default after the build is released. - Make forms POST rather than GET. GET forms are problematic with history cloaking. - Pass css through content filter to fix links, minor adjustment to GenericReadFilterCallback to enable this. - Fix links in filtered HTML, CSS. - Fix lots and lots and lots of links and Location: redirects, may have missed one or two... - Pass LinkFixer in to useralerts. - Pass in original URI to fproxy.handleDownload() so can get the secureid right (freenet:u...@... vs u...@...). - A few things will need to be done *after* this build is released e.g. plugins. - POST to the search form if history cloaking is enabled, otherwise GET for comaptibility with older XMLLibrarian plugins. Web interface: - Separate the search box from the bookmarks box. Tell the user if it hasn't loaded [yet]. - Make the visit-a-plugin button into a link on the classname. - Re-order menu slightly, slight tweak to navbar in clean theme. Bandwidth collapsing bug: - Record info on a UIDTag for every request, insert, or offer reply we send. Keep a map of uid to UIDTag rather than just a set of uid's. Poll every minute, if a request takes longer than 10 minutes, remove it and log as much detail as possible in an ERROR message. - This should workaround the bandwidth collapsing bug and eventually give enough information in ERROR logs to fix it. - Fix a small leak of fetch offered key SSKs. L10n: - Partial German update. - Italian update. - Partial French update. - Don't use getLocalizedMessage() in the filter. It is REALLY slow and we don't have localised exception messages anyway. - Initial commit of Help toadlet (/help/). Not much content yet. - Move a space from the beginning of an l10n string to the code. - Fix typo. Client layer: - Pad the last block BEFORE inserting. We've had four different padding algorithms leading to invisible data corruption, this is a permanent solution. The file will be truncated. Content filter: - HTML: Use a buffered input stream. - CSS: Ignore case on url(), can be URL(). - Fix a filter NPE, not sure whether it was introduced in this build. URI handling: (mostly GenericReadFilterCallback link filter) - RFC-conformant # fragment handling. - Fix bug #2496 - Horrible hack for #2451 - Don't need to trim the key type. - Combine all stripping in FreenetURI(), speeds up constructor by 15%, accept http://.../freenet:c...@... as well as http://.../c...@... - Link filter: See history cloaking, plus encode type override and fragment and uri's in HTTP escapes. Connection setup: - Wait for the ack when disconnecting from an opennet peer we don't want. - Remove an unnecessary (impossible) length check. - Remove support for broken neg type 3. Minor optimisations (from FindBugs): - Use number.valueOf, not new number() for Short, Integer, Long, Byte. - Remove unused fields. - Make classes static. Minor optimisations: - In grabAcks, don't remove one at a time, faster to clear() at end. - Short-cut common case regex in HTMLNode for faster checking of valid node names. - Save memory in TempBucketFactory, MultiReaderBucket: most buckets only open one read stream/read bucket at once. FCP: - Return all peers status, not just darknet peers. No point sorting as we only count statuses. Show opennet size estimate. Show number of seed servers and clients. - Don't sort peers list. Crypto: - Add nanoTime in to initial seeding for Yarrow as well as the rest. Dev stuff: - Paranoia: Can't dropHostname() if _address is null, don't lookup if it is. - Rename KeyTracker to SessionKey. - Remove unnecessary COMPRESSOR_TYPE argument in makeMetadata() on SingleFileInserter. - Fix typo toACIIString, keep old version around for plugins but deprecated. - Comments/javadocs. - Document why we null out fields in BloomFilter. - Generics. - Trivial: no else needed after if(blah) throw(). - Class to detect a blacklist of wierd characters, or to detect unicode linebreaks. - Minor code simplifications. - Cache FreenetURI.toString(). - HTMLEncoder.encodeXML(). - Indenting. - Trivial refactoring, reduce code duplication - Lohhinh. Plugins: - Deprecate FredPluginHTTPAdvanced. - Force close the plugin jar when unloading. - Force classloader for plugin, should fix classnotfoundexception in db4o/perst using plugins. - Provide a function for waking up the TransferThread. Trivial optimisations - Break out of a loop when found the needed data. - Replace key type string with built-in one if equals, rather than interning. - Remove <const string>.length(). - Find char, not string. Unit tests: - Tests for RFC-conformant # fragment handling. Ext #27: - New db4o version, 7.4.71.12224. - Not yet deployed. sdiz p0s tommy juiceman Luke771 nextgens sich Website: (esr, nextgens, toad) - Fix a typo. - Freemail ports in use troubleshooting procedure. - Freemail fetchmail recipe. - Freemail page: Use the web interface to create an account, it's much easier. - Freemail: fix typos. - Improve wording. - resolve bug #2705: http://freenetproject.org/oldtools.html not available - Update bank balance. Installer: (toad) - REMOVE THE FIREFOX PROFILE. It was causing severe problems for a minority of users, and the obvious hack (polling profiles.ini while the browser is running) is probably not a good idea. - Use Firefox if available on Windows, because IE is insecure for Freenet. Tell the user this if we have to use it. - Fix typo causing hiding the user from the logon screen to sometimes fail. Thanks Zero3. JSTUN: - Don't finish detecting if the node has started to shutdown. - Rename DiscoveryTest to DiscoveryTest_ so it isn't treated as a junit test, and delete some other unused files. UPnP: - Set a socket timeout, prevent UPnP from stalling shutdown. Fairly sure the timeout won't cause problems... KeyExplorer: - Workaround for an NPE. XMLLibrarian: (sdiz) - Make a versioned plugin using FredPluginVersioned. - Generics. - Drop new String(string constant). - Fix fd leaks. - Remove dead code. - Remove many global variables. Cleaner code, and will make running multiple searches in parallel not break in wierd ways. Including one that was supposed to be a constant! - Allow searching local indexes as well as inserted indexes. - Remove String.toString(). - Allow multiple spaces between search terms. - Use addAll and retainAll. - Indenting. - Fix searching for words of less than 3 characters. - Factor out LibrarianHandler and URIWrapper to top-level classes. - Rename variables, methods. - Minor optimisations. - Slightly more robust parsing (allow null fileuris). - Add final keyword etc. - Remove duplicated code. - Indenting. - Allow missing trailing / in index site. - Set index to edition 17 of wAnnA via SSK. Later editions not fetchable due to a metadata/client layer/compression bug. - Support both GET and POST forms, for compatibility with older nodes. The form shown is a POST form. XMLSpider: (sdiz, saces) - Create an index only when asked to, and reorganise synchronization because of this. - If a page is refetched manually, delete the old index data for it. - Factor out forced requeueing of a page even if already fetched when the user adds one manually. Use the same procedure when we find an edition of a USK. - Start requests when found an edition. - Use addChild("#", <text>), not "%". - Turn off cascadeOn<anything>. - Fix more db4o-related bugs. - Synchronization fixes. - Don't parse while an index write is about to happen. Show it scheduled, don't run it for 3 minutes after the user asks. (why?) - Cache Term's and TermPosition's in memory. - Limit the number of parse threads running. - Lots of optimisations. - Run the filtering, fetch failure callback and the index construction on the same thread. This is faster and reduces contention. - Shutdown properly. Rollback transactions, shortcut quite out from callbacks etc. Custom exit hook, remove on unload, causes spider to shutdown when the node terminates. - Store maxPageID explicitly. - Use NativeThread's. - Db4o classloader (no idea why, maybe because it is a plugin?). - Commit less often. - Refactoring. - Split out IndexWriter. - More status info. - Pass Term's around directly, don't lookup by md5. - Many many optimisations. - Get rid of some warnings. - Allow multiple spaces between search terms. - Indenting. - Fix L10n bug. - Rename MyCallback -> ClientGetterCallback. - Allow more entries in subindexes, subject to size limit; will slow down index generation somewhat but indexes will be more consistent in size. - Lots of work on unloading / shutdown. - Reduce number of retries to 1. - Separate Config class. Store in db4o. Pass in as parameter. - Config page to change config options. - Separate web interface into separate package. - Add bookmarks when starting, not on every request. - Cache pending requests. - Try to trace a db4o bug. - Mark as succeeded only when it really has succeeded. - Don't intern words. - Cache all TermPosition's while parsing, delete them when the page finishes. - Handle UnsafeContentTypeException parsing a page. - Rollback the database on a serious error. - Port the whole thing to PERST : Less disk i/o, faster processing, lessor CPU, messier code - Various Perst tweaks. - GC the database when making index. - Copy String's in words instead of interning (above), save memory. - Improve locking using rwlocks. - Minor optimisations. - Remove some debugging code. - Logging. - Set classloader on new threads in XMLSpider. - Change method permissions. - Use some exclusive and some co-operative transactions. - Indent. - Always get root from storage so that transaction rollback works properly. - Start immediately on startup. - Throw a RuntimeException on various errors. - Debug info in the stored index if a debug flag is enabled in the config. - Command-line index writer support, opens database read-only, add number of runs parameter to run a benchmark. - Fix a nasty bug causing important data to never be stored (since the change to Perst). - Allow requeuing URIs manually. - Memory optimisation writing indexes, don't pass around lists of all terms, search from the prefixes. - Detect too-big before actually writing the subindex. - More minor code cleanup. - Make startSomeRequests() a callback. - Backport a deadlock fix from unreleased Perst 405. - Set page as failed if it throws an exception when parsing. - Simplify: Config: Use overridden Object.clone(). - Make config page actually work for setting settings. Check the blacklist for valid extensions. - Fix bug #2780 : Update extensions blacklist, check case insensitively. - Hack to reduce the number of metadata files. - Save CPU: Start requests only when running goes below 80% of target. - Re-enable 30 seconds wait on startup. DB4O: (toad,TheSeeker) - Merge trunk up to 24445. - Various merge-related fixes. - Remove data structures when finished with them: done for requests, not done yet for inserts, will need to test properly by running requests and removing them and then iterating the database. - Post-merge fixes e.g. RealCompressor in InsertContext was causing Node to be stored! - Lots of bug fixes! - Lots of activation fixes. - Don't store FCPClient's or ClientRequest's unless they are persistent. - Lots of debugging code - logging, assertions, paranoia etc. - Refactoring. - FCP: Separate RequestClient from client. Connection-persistence requests DO NOT HAVE A CLIENT. Don't send AllData for persistent requests. - Generics etc. - Log an error and fail when trying to store some key objects e.g. Node in the database. - Handle stuff being deleted when we were expecting to deal with it in e.g. DatastoreChecker. - Rename variables. - Delete dead code, dead variables, dead arguments. - Another db4o duplicate objects bug workaround. :( - Priority tweaks. Priority of FEC callbacks depends on request priority. - Copy stuff where necessary to ensure that only what needs to be deleted is deleted. - Indenting. - Comments. - Various bugfixes in SegmentedBucketChainBucket, related to freeing data (uses a blocking db job when necessary, avoid an OOM...). - Startup fix: Make sure we init ArchiveManager first, and run the startup jobs really early. - Fix infinite loop in copying from a blob bucket. - Some work towards fixing a major space leak in the blob temp file system. - RegisterMeRunner is only needed for inserts. - Pad the last block BEFORE inserting. We've had four different padding algorithms leading to invisible data corruption, this is a permanent solution. The file will be truncated. This has been backported to trunk in SVN r24997. - Caller must pad if necessary: FECCodec no longer supports padding the last block, SplitFileFetcherSegment or SplitFileInserterSegment must do this itself for both inserts and requests. Modified: trunk/freenet/src/freenet/node/Version.java =================================================================== --- trunk/freenet/src/freenet/node/Version.java 2009-01-10 22:24:07 UTC (rev 25016) +++ trunk/freenet/src/freenet/node/Version.java 2009-01-10 22:29:24 UTC (rev 25017) @@ -24,17 +24,17 @@ public static final String protocolVersion = "1.0"; /** The build number of the current revision */ - private static final int buildNumber = 1194; + private static final int buildNumber = 1195; /** Oldest build of Fred we will talk to */ - private static final int oldLastGoodBuild = 1193; - private static final int newLastGoodBuild = 1194; + private static final int oldLastGoodBuild = 1194; + private static final int newLastGoodBuild = 1195; static final long transitionTime; static { final Calendar _cal = Calendar.getInstance(TimeZone.getTimeZone("GMT")); // year, month - 1 (or constant), day, hour, minute, second - _cal.set( 2008, Calendar.DECEMBER, 19, 0, 0, 0 ); + _cal.set( 2008, Calendar.JANUARY, 12, 0, 0, 0 ); transitionTime = _cal.getTimeInMillis(); } _______________________________________________ cvs mailing list [email protected] http://emu.freenetproject.org/cgi-bin/mailman/listinfo/cvs
