On Thursday 23 April 2009 14:36:37 xor wrote: > > > -----Original Message----- > > From: devl-bounces at freenetproject.org > > [mailto:devl-bounces at freenetproject.org] On Behalf Of Matthew Toseland > > Sent: Thursday, April 23, 2009 3:17 PM > > To: support at freenetproject.org; Discussion of development issues > > Cc: Ian Clarke > > Subject: [freenet-dev] Solving "I queued it 2 weeks ago and > > it's still at0%" : are really long URIs a problem? > > > > Anecdotal evidence suggests that right now at least one third > > of our content persistence problems boil down to this one > > bug: "I added it 2 weeks ago and it still hasn't got past 0% > > (0/1)". A new key type, DHKs (Duplicated Hash Keys), would > > solve the problem, but the new keys would be twice as long as > > current CHKs. Is this a problem? I would really appreciate > > input from users, particularly those who upload and download files: > > - Is it a problem for the keys to be really long (twice as > > long as current CHKs)? CHKs are copied and pasted, so maybe > > not a problem? > > - Is it true that a great many downloads get stuck at 0% for > > a long time, showing 0 blocks of 1 if you mouseover the percentage? > > I think you should first of all figure out whether it isn't a db4o > introduced bug or even a problem which has been there before db4o.
AFAIK it is a general problem with Freenet, that predates db4o. To be specific: - Most downloads of older data stall at 0% for a *long time*. - When they get past 0%, they frequently progress fairly quickly. - Splitfile blocks have redundancy, but the top block does not. - Unpopular (old) content will have fallen out of caches. - Unpopular content will only be in the stores of the small number of nodes (typically 3 but a bit more because of not taking into account low-uptime nodes). - Low uptime and join-leave churn (newbies trying Freenet out and then leaving) mean that there is a good chance that all of these nodes are offline when you want to fetch the content. It is also possible that they have shifted location. > Relying on something brand new (the db4o branch) which has been out in the > real world only for some days to decide implementation of a new key type is > really insane IMHO. We don't know whether the db4o branch doesn't have > hundreds of other bugs. And in general the debugging of the node has been > mostly on hold for long time due to your work on db4o. I have done a vast amount of debugging on db4o while it was a branch. I accept that it likely has many bugs. However we have approximately 34 days to get a release out before we run out of money. > > This would not be the first bug which makes downloads hang at 0% which we > had. Sure, it might well be a bug, in which case we get a better architecture (everything except the top block is redundant already), we spend a few days implementing and debugging DHKs, and we still have to solve the bug. > > Further I do not understand why you don't want to solve the problem with > tunnels. You told me that tunnels fix three problems: Tunnels will not solve the problem if low uptime, newbies joining and leaving, and location swapping on darknet, are the root of the problem. Everything is redundant below the top block, and below the top block it works relatively well. So add redundancy for the top block. More redundancy for *all* blocks would be counterproductive, because block level redundancy is far less efficient than FEC redundancy, because one block can be reconstructed from other blocks and healed during a download. The top block is a special case because it can't be FEC'ed. > > 1. they increase security True. > 2. <toad_> well the other benefit is that it allows us to do bloom filter > sharing [17:07:58] Yes but the performance cost is considerable. > 3. tunnels would fix the top-block-not-reachable problem we are talking > about No, they would not, as I have explained. > > So tunnels would fix 3 problems, this fixes 1 and tunnels will still be > necessary some day. > > Why not just postpone the fix of the top-block-problem until we have > tunnels? Tunnels are awesome, but they will be take quite some time to get right, and we really have not yet quantified the performance impact of bloom filters plus tunnels. It's a 0.9 matter. Right now we have funds for me for approx 34 days. > > Why not make the node just automatically re-insert the top block upon > download? Thats easy to implement after all? It does not introduce any additional redundancy. We need redundancy across *different areas of the keyspace*. > > Why not debug first? We will debug. But IMHO this is an architectural problem, as nextgens has pointed out. > > I don't understand why you would want to chose the annoying alternative of > implementing yet another key type if there are so many alternatives to it. Because none of the alternatives solves the problem. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 835 bytes Desc: This is a digitally signed message part. URL: <https://emu.freenetproject.org/pipermail/devl/attachments/20090423/f1f709b7/attachment.pgp>
