On Wednesday 14 Mar 2012 11:49:33 Ximin Luo wrote: > Library has some architectural problems. I don't think it's a great use of > time > to try to improve the existing code directly. > > - We didn't create a proper specification for how the data structure should > work over freenet > - I made a mistake in trying to shoe-horn a stub remote data structure into > the > Java collections framework. > - To support loading from remote, I wrote my own event handling framework, > which was not a very clean design as I wasn't familiar with > java.util.concurrent at the time. > - I did write an asynchronous merge algorithm (SkelBTreeMap.update) which > currently does the majority of the write work, but it's quite complex and > doesn't integrate well with the rest of the code. Toad also made some > adjustments on top of it for performance, which makes understanding it even > more complex as it distracts from the "pure" form of the algorithm. > > What is needed to get Library working well is to: > > 1. specify the data structure, and associated algorithms, properly. I have > notes, I can help with this > 2. use an event handling framework for Java (something similar to Twisted for > python, but also has the ability to use threads if necessary) > 3. use this to implement the algorithm in a concise way
Some functional stuff for Library (related to on-network format), that may or may not fit in depending on how much is being thrown out and rewritten: - B+tree (no data in the nodes so reduced depth), not B-tree - Do we need any on-network changes to limit memory usage? Maybe consider how to deal with term positions... - Some optimisations were proposed that might allow for only fetching a single block in the common case (we'd fetch 2 others for redundancy as well); this would probably require a binary format and might run into metadata size issues? See the bug tracker anyway. https://bugs.freenetproject.org/view.php?id=4066 And categories: library b-tree-index Also, there is another search plugin, it doesn't scale but has a very compact format and is very fast. Worth having a look and/or reaching out to its anonymous author? I dunno what the current status is. I never got around to making it an official plugin but the rumour was that it worked well. It's a shame not to bring in an anonymous contributor who goes to the trouble of writing a working plugin... > > freenet itself should use something like this as well - atm it uses its own > custom solution which is a bit messy and not well-separated from the rest of > the code. For what exactly? I'm not sure I follow? One thing we MUST MUST MUST be able to do is *WRITE* the config file. Obviously this is optional, and on some unixy setups with a competent admin we can run without the ability to write the config file; but for 99.9% of users the ability to configure Freenet in a GUI way is useful, and from time to time it is helpful to be able to update config settings from e.g. a useralert. > It also uses many of its own crypto and concurrent utilities, which > bloats the code for no reason. True. I have no problem with replacing them with the standard stuff provided the impact on performance on the common case of no hardware acceleration is positive or negligible. (The API is a bit messy). Also, are we sure the JVM export keylength BS is finished, over and gone? One major complication is quite a lot of Freenet uses 256/256 AES which is sort of nonstandard, and probably not going to be hardware accelerated. Replacing it would probably involve changing some link-level formats e.g. so we could use CBC. Likewise PCFB is fairly widely used, which is also somewhat nonstandard, although with 256/256 it is identical to CFB, which is more common. Let me know what you need, but I'm not going to do all the work for you. We have plenty of back compatibility related stuff in place - autonegotiation of connection type, version numbers on keys etc. > > In other words, both Library and freenet (probably freenet first) needs some > major refactoring work, before major "features" can progress. No, I disagree. Refactoring can happen at any time, if a new major feature depends on it then it should be expedited. However I'm happy to accept your view re Library; re Freenet, I'm happy for other people to do refactorings they see as important, but I don't have time (or funding) for anything large myself. > > Most of this requires that freenet-ext be available as separate modules, which > needs an overhaul of the update-over-freenet system as that currently assumse > freenet-ext comes in a single JAR. Yes, this is important. The tricky part here is it would affect *everything*: The installer, the manual update scripts, the updater ... Let me know what you need. > > Oh, and another thing, the config framework is horrible and we really should > use an existing external solution. Possibly. It works; what are the main problems as you see it? I'm certainly open to ripping out custom code and replacing it with standard libraries; even third party libraries if they're not going to massively bloat the download, although they do *increase* maintenance costs... > This would make the freenet installation > portable to different systems (e.g. to conform to FHS), rather than being > constrained to the current rigid set of directories. I did make this part > customisable but it's still untidy and has some fiddly non-fatal problems. Well, Freenet is a daemon. You can't treat it as a utility that has each user's data in their home and the program files in /usr. At the same time, unless it is packaged, it needs to be able to update its own binaries. Both of these conflict with the Standard Unix Way Of Doing Things. > > Dependency diagram: > > [ overhaul freenet update system ] This is the most urgent/important thing: Split up freenet-ext.jar and make the installers, manual update scripts and internal auto-update code able to deal with this. Most of this is fairly straightforward but it requires a variety of skills and has to be deployed all at once ... Beyond that we've discussed other possibilities e.g. using a single script to deploy downloaded jars from both freenet and the manual emergency update from the web ... > |-> [ use external crypto library ] This is a good thing, but there are complications as mentioned. > |-> [ use external config management library ] I am yet to be convinced that this is a big gain. > | |-> [ make installation more portable / robust ] I would be interested in specific examples. Be careful not to be too unix-centric. > |-> [ use external event-handling library (both async and concurrent) ] I'm not sure exactly what is involved here, apart from making Ticker and Executor use the standard API's. (While not losing the ability to use PrioRunnable's and NativeThread's to tweak the thread priorities). If you're talking about client.async, well, that's a whole other email (which I'm not gonna write right now) ... > | |-> [ update the Library plugin as described ] > > This is the ideal non-hacky correct way of doing things, but I gather we don't > have that much development time on the whole. That depends on you, and other volunteers :) "Toad helps and does things Ian/FPI think are important", not "Toad does everything". > > X > > On 14/03/12 05:06, Juiceman wrote: > > Hi, I think you wrote or worked on the "Library" plugin for Freenet? > > I posted a job at > > http://gun.io/open/45/update-freenet-plugin-library-to-snakeyaml-110 > > for someone to update the code, figured you might be interested? The > > bounty is only $25 USD right now, I wish I could afford more.... > > > > "The Seeker" took a stab at it but couldn't get it to work. This is > > what he said to me on Sone: > > > > The Seeker > > @Juiceman I tried doing this, but just making it compile and run isn't > > nearly enough :-/ When I try to do a search, I get this 3-layer > > stacktrace... trying to follow wtf is going on makes my brain hurt. I > > give up. plugins.Library.util.exec.TaskAbortException: Could not > > construct index from data -- > > plugins.Library.index.ProtoIndexSerialiser.pull(ProtoIndexSerialiser.java:123) > > -- plugins.Library.Library.getIndex(Library.java:677) -- > > plugins.Library.Library.getIndex(Library.java:640) -- > > plugins.Library.Library.getIndex(Library.java:616) -- > > plugins.Library.search.Search.splitQuery(Search.java:245) -- > > plugins.Library.search.Search.startSearch(Search.java:111) -- > > plugins.Library.search.Search.splitQuery(Search.java:347) -- > > plugins.Library.search.Search.startSearch(Search.java:111) -- > > plugins.Library.search.Search.startSearch(Search.java:117) -- > > plugins.Library.ui.MainPage.processPostRequest(MainPage.java:192) -- > > plugins.Library.ui.MainPageToadlet.handleMethodPOST(MainPageToadlet.java:96) > > -- sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) -- > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > > -- > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > > -- java.lang.reflect.Method.invoke(Method.java:597) -- > > freenet.clients.http.ToadletContextImpl.handle(ToadletContextImpl.java:564) > > -- > > freenet.clients.http.SimpleToadletServer$SocketHandler.run(SimpleToadletServer.java:1102) > > -- freenet.support.PooledExecutor$MyThread.realRun(PooledExecutor.java:233) > > -- freenet.support.io.NativeThread.run(NativeThread.java:130) > > plugins.Library.io.DataFormatException: TreeMapTranslator: reverse > > translation failed. Try supplying a non-null key-translator. -- > > plugins.Library.util.SkeletonTreeMap$TreeMapTranslator.rev(SkeletonTreeMap.java:330) > > -- > > plugins.Library.index.ProtoIndexComponentSerialiser$TreeMapTranslator.rev(ProtoIndexComponentSerialiser.java:324) > > -- > > plugins.Library.index.ProtoIndexComponentSerialiser$TreeMapTranslator.rev(ProtoIndexComponentSerialiser.java:310) > > -- > > plugins.Library.util.SkeletonBTreeMap$NodeTranslator.rev(SkeletonBTreeMap.java:1519) > > -- > > plugins.Library.util.SkeletonBTreeMap$TreeTranslator.rev(SkeletonBTreeMap.java:1582) > > -- > > plugins.Library.util.SkeletonBTreeMap$TreeTranslator.rev(SkeletonBTreeMap.java:1551) > > -- > > plugins.Library.index.ProtoIndexSerialiser$IndexTranslator.rev(ProtoIndexSerialiser.java:209) > > -- > > plugins.Library.index.ProtoIndexSerialiser$IndexTranslator.rev(ProtoIndexSerialiser.java:134) > > -- > > plugins.Library.index.ProtoIndexSerialiser.pull(ProtoIndexSerialiser.java:121) > > -- plugins.Library.Library.getIndex(Library.java:677) -- > > plugins.Library.Library.getIndex(Library.java:640) -- > > plugins.Library.Library.getIndex(Library.java:616) -- > > plugins.Library.search.Search.splitQuery(Search.java:245) -- > > plugins.Library.search.Search.startSearch(Search.java:111) -- > > plugins.Library.search.Search.splitQuery(Search.java:347) -- > > plugins.Library.search.Search.startSearch(Search.java:111) -- > > plugins.Library.search.Search.startSearch(Search.java:117) -- > > plugins.Library.ui.MainPage.processPostRequest(MainPage.java:192) -- > > plugins.Library.ui.MainPageToadlet.handleMethodPOST(MainPageToadlet.java:96) > > -- sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) -- > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > > -- > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > > -- java.lang.reflect.Method.invoke(Method.java:597) -- > > freenet.clients.http.ToadletContextImpl.handle(ToadletContextImpl.java:564) > > -- > > freenet.clients.http.SimpleToadletServer$SocketHandler.run(SimpleToadletServer.java:1102) > > -- freenet.support.PooledExecutor$MyThread.realRun(PooledExecutor.java:233) > > -- freenet.support.io.NativeThread.run(NativeThread.java:130) > > java.lang.ClassCastException: java.lang.String cannot be cast to > > java.lang.Double -- java.lang.Double.compareTo(Double.java:32) -- > > java.util.TreeMap.getEntry(TreeMap.java:328) -- > > java.util.TreeMap.get(TreeMap.java:255) -- > > plugins.Library.util.SkeletonTreeMap.putGhost(SkeletonTreeMap.java:95) > > -- > > plugins.Library.util.SkeletonTreeMap$TreeMapTranslator.rev(SkeletonTreeMap.java:327) > > -- > > plugins.Library.index.ProtoIndexComponentSerialiser$TreeMapTranslator.rev(ProtoIndexComponentSerialiser.java:324) > > -- > > plugins.Library.index.ProtoIndexComponentSerialiser$TreeMapTranslator.rev(ProtoIndexComponentSerialiser.java:310) > > -- > > plugins.Library.util.SkeletonBTreeMap$NodeTranslator.rev(SkeletonBTreeMap.java:1519) > > -- > > plugins.Library.util.SkeletonBTreeMap$TreeTranslator.rev(SkeletonBTreeMap.java:1582) > > -- > > plugins.Library.util.SkeletonBTreeMap$TreeTranslator.rev(SkeletonBTreeMap.java:1551) > > -- > > plugins.Library.index.ProtoIndexSerialiser$IndexTranslator.rev(ProtoIndexSerialiser.java:209) > > -- > > plugins.Library.index.ProtoIndexSerialiser$IndexTranslator.rev(ProtoIndexSerialiser.java:134) > > -- > > plugins.Library.index.ProtoIndexSerialiser.pull(ProtoIndexSerialiser.java:121) > > -- plugins.Library.Library.getIndex(Library.java:677) -- > > plugins.Library.Library.getIndex(Library.java:640) -- > > plugins.Library.Library.getIndex(Library.java:616) -- > > plugins.Library.search.Search.splitQuery(Search.java:245) -- > > plugins.Library.search.Search.startSearch(Search.java:111) -- > > plugins.Library.search.Search.splitQuery(Search.java:347) -- > > plugins.Library.search.Search.startSearch(Search.java:111) -- > > plugins.Library.search.Search.startSearch(Search.java:117) -- > > plugins.Library.ui.MainPage.processPostRequest(MainPage.java:192) -- > > plugins.Library.ui.MainPageToadlet.handleMethodPOST(MainPageToadlet.java:96) > > -- sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) -- > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > > -- > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > > -- java.lang.reflect.Method.invoke(Method.java:597) -- > > freenet.clients.http.ToadletContextImpl.handle(ToadletContextImpl.java:564) > > -- > > freenet.clients.http.SimpleToadletServer$SocketHandler.run(SimpleToadletServer.java:1102) > > -- freenet.support.PooledExecutor$MyThread.realRun(PooledExecutor.java:233) > > -- freenet.support.io.NativeThread.run(NativeThread.java:130) > > > > What do you think? > > > > -- > > I may disagree with what you have to say, but I shall defend, to the > > death, your right to say it. - Voltaire > > Those who would give up Liberty, to purchase temporary Safety, deserve > > neither Liberty nor Safety. - Ben Franklin > > > -- > GPG: 4096R/5FBBDBCE > https://github.com/infinity0 > https://bitbucket.org/infinity0 > https://launchpad.net/~infinity0 > > -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 198 bytes Desc: This is a digitally signed message part. URL: <https://emu.freenetproject.org/pipermail/devl/attachments/20120315/c0c35f1c/attachment.pgp>
