On Thursday 16 May 2013 17:47:14 leuchtkaefer wrote: > > Hello everybody, > > I am planning to implement file sharing/search improvements/new features. As > an initial step I immersed myself in all different sources of info about > Freenet to understand better all part of the projects related to the > distributed search. I more or less read lot of stuff listed in a guide that I > started writing in case someone is in the same situation as me in the near > future. I can uploaded my guide to the list if somebody is interested on it. > > > HOWEVER it is still difficult for a newbie in Freenet to understand the basic > implemented data structures. It is also somehow confusing know what things > have been done but are not in the official deployment. > > For instance, I really need to understand: > > > According to doc in source Library an INDEX has metadata and two BTree.Index > |-- metadata > |-- ttab: BTreeMap<String, BTreeSet<TermEntry>> > |-- utab: BTreeMap<URIKey, BTreeMap<FreenetURI, URIEntry>> > > Does each p2p node construct one index? It is this index the actual structure > in official Freenet? > > I also don't understand why the root of the index is stored under a SSK > splitfile. I thought and SSK splitfile is used to store normal user's file. > > THanks for help in advance!
Basically the layering is: Storage layer: SSKs (1kB, named by pubkey and filename), CHKs (32kB, named by content hash) Splitfiles/client layer: A file is uploaded to freenet to a URI (CHK or SSK). The file can be any size. If it doesn't fit into one block, we split it up into a pyramid consisting of one block (a CHK or an SSK), containing metadata (a list of keys) for the next layer. The next layer may be the final data, or it may be metadata for another layer. And so on. Each layer other than the top block has redundancy. Btree nodes: Each btree node is a file (a single CHK block or a splitfile). The nodes are always inserted as CHKs. Btree: Top node, inserted as a CHK like the other nodes. Note that these trees can be updated incrementally, not necessarily by the person who originally uploaded them. Library index: The top USK points to the two trees and has some metadata e.g. the size of the index. We have two trees inside a library index at the top level, and IIRC we use a tree within each term as well to store data by relevance. Currently Spider spiders freenet and uses Library to incrementally upload an index. This is for *searching HTML*, i.e. freesites; currently we don't use this for filesharing.
signature.asc
Description: This is a digitally signed message part.
_______________________________________________ Devl mailing list [email protected] https://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl
