You raise very excellent points - I am glad someone took the bait and is willing to flesh this out with me =)
> Who controls the spider program? Who are they, and > why should > we trust them? Freenet's stability and power is in This becomes a mater that is very very similar to The Freedom Engine. There is only 1 freedom engine and everyone trusts him to post links to everything that is given to him. You don't have to trust TFE but you don't have to use it either - I view this the same way. If you want to actualy get a search done use this system, if you want to be 100% sure that you don't trust anyone don't use it and get your keys only from trusted sources. I am willing to run the entire system from my machine - I am a professional unix administrator and architect. I would also setup the search system so that it is highly portable and have plans that existed to facilitate moving the system to another machine either onsite or offsite in case something goes wrong here. I would also be more then willing to publish everything possible - that seems it would be everything except the private key for the SSK where the search engine lives in freenet. > Well, I assume that the spider would end up > filtering common > words out of pages, otherwise you'd end up inserting > more than > 200,000 keys into this 'index' subspace since there > would be > many, many, many unique words. > > But what if I want to search for 'TEST'? or 'tESt'? > What if > what I'm specifically looking for is ' test'? > (With two spaces) > If you upload the indexes case insensitive, then > that kinda screws > the people who need case sensitivity. If you upload > the indexes > case sensitive, that screws people who need case > insensitivity. If > you upload both, well then it's good that you're > only proposing indexing > once every week, because it's going to take that > bloody long just to > insert all of the keys. I had envisoned only using case insensitive search. I could not imagine uploading every word that exists with mixed case - I would have the indexer lowercase every word it finds to help flatten the space out a bit. While this is not 100% favorable it is consistent with real world search engines. I personaly do not know of a large scale search engine that will respect a mixed case word that is thrown at it - everyone I have seen is completely case insensitive. The system would definitely require stop-words to help keep the number of indexs down. If we really really really wanted to be evil we could limit the total possible keys to around 25k (or some other arbitrary number, this just happened to be the size of the dictionary on my machine) by only allowing words that appear in a dictionary file. This would only work for english but nothing is stopping us from setting up another section for each language with it's own dictionary file. > I think it's a good idea if you can find somebody > that everybody can > trust, which you can't. People don't even trust the > authors of the > software. If the index were compromised, then the > new operator could > put whatever he wanted in there, which could be > potentially very bad. > > For a localized freenet network where having one > central authority is > acceptable, I think this strategy would work fine. > But for the wider > public freenet where no one trusts anyone else, I > think you'd have a > hard time selling this. Again, the entire point of > freenet is to put > the control of the network in the hands > of....nobody. Somebody has to > have that public/private keypair for the index SSK. > Somebody has to > run the spider and upload the requisite keys for > searching. Building > the trust in that somebody is pretty much > impossible, since even if it > were Ian Clarke, he would be vulnerable to takeover > by forces who don't > want freenet used, or he could simply be thrown in > jail. > > Any search idea that will be acceptable to the > public anonymous network > is probably going to have to be controlled by > nobody, like the rest of > the network. And that's really hard - which is > probably why we don't > already have a search mechanism. :) Again, I refer you to TFE. With out TFE freenet would be much more difficult to navigate around in - you would be stuck getting your keys only from word of mouth - whether that be meatspace or frost/FMB you still don't have a nice convienent announcement spot. The amount of trust required to do searches is very very very minimal. It is not like this design requires you to execute code on your local machine that is coming over freenet. If the developers put it into fproxy you would only have to trust it as much as you trust the rest of their code. If someone writes a stand alone tool you have to trust them too - but if you run frost you have to trust the author of that application, or the author of FMB. We should also envision the worst case scenario - the private key is compromised and an attack now uploads his own arbitrary indexs. This is actualy not that big a deal. Honestly, its not like this is going to cause a large scale failure of the freenet system or going to facilitate compromising individual people's nodes. At the worst the attacker can now refer people to arbitrary freesites that do not actualy contain the word the user searched for - not a big deal. This is a problem that any freesite must deal with, TFE included. If you would like to argue the point of trust I ask you to apply each argument to TFE. If there is a point that separates the amount of trust required for a client to use my design that is different then the trust required for a client to use TFE I am willing to hear it. Seeing as both of them only request keys out of freenet then use code executing on the local machine provided by a trusted source to deal with the data and will ultimately display it to the user I really don't see that big a difference. I really really thank you for attacking my design =) It is the only way to solve all the problems before the thing rolls out (if it does). I understand that this is not a 100% ideal solution but as you said if we wait for the 100% ideal solution to come we could be waiting forever. This would allow search to be done with in a couple weeks and available to the public. Tyler ===== AIM:rllybites Y! Messenger:triddle_1999 __________________________________________________ Do you Yahoo!? New DSL Internet Access from SBC & Yahoo! http://sbc.yahoo.com _______________________________________________ Tech mailing list [EMAIL PROTECTED] http://hawk.freenetproject.org/cgi-bin/mailman/listinfo/tech
