On Wednesday 20 January 2010 10:37:12 alex wrote: > Matthew Toseland wrote: > > > 2) Write a killer file-sharing application (232 votes) > > > > IMHO there are a number of issues here: > > - Current data persistence sucks. We are working on it and need to do more > > work on it. - It is relatively hard to search for files. We need a good > > WoT-based spamproof fast file search system. - General demand from many > > folks for insert on demand. Maybe it worked on Frost, maybe it didn't, but > > Frost is dead. The main advantages of insert on demand are 1) that you can > > "share" a directory immediately, 2) you don't need to wait for the insert, > > 3) the set of available data, or available data at a given speed, is > > larger. I am skeptical, there are several problems with it (trust, > > security, different views of "network pollution" ...), but in any case > > improving data persistence and making it easy to search for files are both > > important, and once we have these we can consider insert on demand. And > > there *are* options for relatively safe reinsert on demand, although they > > are messy... > > > > Or am I being unduly dogmatic and long-termist here? Maybe a simple > > WoT-based file search and share and reinsert on demand system would gain > > us so many more users that we should do it anyway, even if insert on > > demand is risky and not necessary long term? > > > > This does rely on distributed searching, and it could be a while ... > > This is only MHO, but I really think that freenet could take off on user > numbers if it were an out-of-the-box secure file-sharing replacement. It's a > matter of time that some app will fill the gap that will appear, even if > less private alternatives are nowadays doing well.
"Out of the box" is a challenge. Any sort of distributed search is not going to work instantly. However, Perfect Dark takes a couple of hours to download the indexes, so maybe it doesn't have to? Opennet needs to work well out of the box and we need to do more work on that: Dealing with low uptime well is important. The fact that you can insert a popular freesite from a cafe with wifi is a major benefit in terms of features and use cases, we should play to it. > > Backing this position are emergent projects aiming at providing shared > global hard-disks, like omemo, wuala and others (you could even include > dropbox and the myriad of similar ones here, although their sharing > capabilities are not (I think) intended for untrusted people groups). In a > way they're doing what freenet does in providing free "cloud" storage. Wuala uses 517% FEC redundancy plus central backup servers. IMHO we will need something like that level of redundancy (although take into account that we have some block level). No idea about omemo, beyond what wikipedia says: it's an open source storage DHT with randomisation. > > The case of omemo is of particular interest to me, not only because his > creator is a compatriot, long time prominent figure in the p2p scene, and > awaiting sentence for *creating* a p2p application (actually the charges are > unfair competition, worth 13e6$), but because his programming trajectory > follows what I see as the dominating trend in p2p generations, in going from > mp2p to omemo. Ooops! > > So better be there leading than lose the train to some not-so-secure > alternative. Not so secure alternatives will be faster for a long time, and may have commercial muscle behind them and therefore better UIs. We will not be there before them. But we may eventually outcompete them for some markets - people who really care about privacy, for example. > > That said, I'm with you in that several pieces are needed, and all of them > require lots of work (even in theoretical aspects). > > a) Data persistence Lots of work needs to be done here. Much has already been done. And this is data persistence in the broadest sense - it incorporates how fast you can find rare or old data as well as whether you can find it at all. > b) Distributed scalable spam-resistant indexing (!) This is hard. I will try to explain the technical issues. infinity0's main, more or less complete contribution, has been the new index format. This is essentially an on-Freenet btree (a standard structure used in databases), which means it scales to any size without any of the parts getting enormous. Whereas the current index format has problems when it gets big. The first serious application of it will be to replace the current spider format, and write the data on the fly while indexing rather than having to wait for a week while the XML format data is written. It will also include the actual insert to Freenet, unlike the current code, will support updating an on-Freenet index without reinserting everything, and generally will be a huge gain. It is necessary to have a scalable structure for two main reasons: 1) Searching freesites more or less requires a spider. This will put out huge indexes. 2) Distributed searching will likely involve some users aggregating other users' indexes, so even for file searching, big indexes will be needed eventually. Now, for full distributed searching, we need more work. Most of that (Interdex) has been specified, planned and designed in some detail by infinity0, but it will not be ready in the near future as he has no time. It is conceivable that others might finish it. Basically it will be a WoT-based search system where each user publishes an index, and also links to other people's indexes. It will be relatively slow, because it involves fetching lots of data, but we can preload, like Perfect Dark does. And then you have scaling issues with the WoT based search itself. Although I am of the view that scaling issues on Freenet are not quite the same as you might expect: Fetching popular data is easy and produces very little network load. However if the set of indexes grows very large, life gets difficult. This is a problem for centralised searching (like the current freesite indexes) and even more so for decentralised searching... > c) Insertion on demand. > > Also like you I wonder if it's better to get a working something ASAP and > refine it later or what. The frost system death should be a warning, but... It's a question of what you mean by "working". Once we have any form of distributed search, we will have something that works, after a fashion, even if we do not have insert on demand. We could then implement a quick hack for insert on demand, but I'm not sure whether that is the right approach given serious long-term issues with insert on demand which I am not sure how to solve... There are several serious problems with insert on demand: 1) Insert time security. If the attacker knows what data you are going to insert in advance, it makes some very serious attacks *much* easier. 2) Uptime issues. The data will only be inserted if somebody who has it is online. Thus it can take days, and response time patterns of inserters may give away their location etc. 3) DoS issues. How do you prevent bogus requests for data in an automated insert on demand system? Maybe WoT will help, but it's early days. 4) Capacity. IMHO if Freenet is working well we should not need insert on demand: Its capacity should be much greater than it is now, and we should be able to just insert and fetch the data. Most of the obvious solutions for the first three issues end up breaking one of the other three. There are some radical options for security, but these generally involve sacrificing predictable keys, meaning the requestors may have to wait for the insert to be complete, or at least waste space by inserting it to a different key and then have the requesters possibly reinsert it to the standard location. > > I'm awaiting eagerly for the WoT/Freetalk plugins to be deployed, although > my (distant) experiences with FMS WoT weren't that rosy. > > I get lost in the technical details, so I wonder if some recent posts about > insertion of b-trees can help b), which to me is the most difficult goal... I have tried to clarify that above. The answer is yes, it is closely related. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 835 bytes Desc: This is a digitally signed message part. URL: <https://emu.freenetproject.org/pipermail/devl/attachments/20100120/f7bc5c62/attachment.pgp>
