On Wednesday 20 January 2010 15:44:22 Evan Daniel wrote: > On Wed, Jan 20, 2010 at 8:54 AM, Matthew Toseland > <toad at amphibian.dyndns.org> wrote: > > > 4) Capacity. IMHO if Freenet is working well we should not need insert on > > demand: Its capacity should be much greater than it is now, and we should > > be able to just insert and fetch the data. > > Actually, I'm not completely convinced this is a problem. At present, > I believe most of our data persistence issues stem from node churn, > not blocks falling out of individual stores. Right now that's just a > (somewhat justified) hunch, but I have sufficient data to investigate > in more detail.
Okay, lets consider your old data (from flog): == That says to me that approximately 38% of users are "occasional" users, who either only run their node some of the time, or install, run briefly, and then uninstall. A further 23% are dedicated users ? they have their nodes on all the time. The remaining 38% (in 1 or 2 samples) I'll call "regular" users ? they frequently have their node running, but not always. Obviously, these classifications are very rough. I'd say a 1:2:2 ratio is probably a reasonable guess, but it could still be rather far off. I need to take data more regularly and for a longer period of time before any serious conclusions can be drawn. However, I am comfortable saying the following: Freenet has at least 4000 semi-regular or regular users (probably meaningfully more). Freenet probably has between 8000 and 12000 total users (the upper bound I'm far less certain of ? if a lot of people only run Freenet for an hour or two per day, it could be far higher). At most, about a third of users run their node 24/7; the actual number is probably well under that. I think this has several practical implications. First, we need to be working on data retention more, with a focus on retention despite low-uptime nodes. (See bugs 3495, 3514 for a start on that. 2933 should also help. 3637/3639 and the like address more general routing issues; that should help as well.) Second, we need to figure out how to get these low-uptime nodes back onto the network, and connected usefully, so that the data they have can be found (and to improve the performance for such users). (See 3583 and related bugs for one approach.) And, finally, we have the general problem of getting (and keeping!) more users. == So lets say it's reasonable to assume that 40% are 24/7, 20% are newbies, and 40% are maybe an average of 50% uptime? Can you give a rough figure for the average uptime in the "regular" group? Clearly 3639 needs to be fixed. Most of the other bugs you mention have been fixed already. However, what it comes down to really is we need more (or better) redundancy if we want stuff to be available immediately (after it has been on the network for long enough that it is only in stores and not caches). That means: - Considering more store-level block-level redundancy. IMHO we have largely exhausted this once we have fixed 3639: We don't want excessive block-level redundancy because there are other options which are better. - Considering more or better splitfile-level redundancy. Fixing the splitting problems, possibly increasing the FEC codes, possibly introducing an interlocking code as on CDs. Wuala uses 517% FEC and they still have to have backup servers in practice; we have block level redundancy, but it's still worth seriously considering ... - More or better top block redundancy. I need to look at the MHK tester results: Is inserting the same block 3 times better or worse than inserting 3 different blocks? See my other mail! IMHO improving data persistence is *THE* way we are going to get Freenet to the point where it is really usable. It should be a priority, although clearly we can't do everything before 0.8.0. Of course there are other issues - ease of use, fixing filesharing, etc. But IMHO make it work and they will come. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 835 bytes Desc: This is a digitally signed message part. URL: <https://emu.freenetproject.org/pipermail/devl/attachments/20100121/c45e85f5/attachment.pgp>
