On Thursday 24 June 2010 18:28:42 Matthew Toseland wrote: > On Thursday 24 June 2010 04:05:48 Evan Daniel wrote: > > On Wed, Jun 23, 2010 at 5:43 PM, Matthew Toseland > > <t...@amphibian.dyndns.org> wrote: > > > On Wednesday 23 June 2010 20:33:50 Sich wrote: > > >> Le 23/06/2010 21:01, Matthew Toseland a écrit : > > >> > Insert a random, safe key > > >> > This is much safer than the first option, but the key will be > > >> > different every time you or somebody else inserts the key. Use this if > > >> > you are the original source of some sensitive data. > > >> > > > >> > > > >> Very interesting for filesharing if we split the file. > > >> When some chunk are lost, you have only to reinsert those who are > > >> lost... But then we use much datastore... But it's more secure... > > >> Loosing datastore space is a big problem no ? > > > > > > If some people use the new key and some use the old then it's a problem. > > > If everyone uses one or the other it isn't. I guess this is another > > > reason to use par files etc (ugh). > > > > > > The next round of major changes (probably in 1255) will introduce > > > cross-segment redundancy which should improve the reliability of really > > > big files. > > > > > > Long term we may have selective reinsert support, but of course that > > > would be nearly as unsafe as reinserting the whole file to the same key > > > ... > > > > > > If you're building a reinsert-on-demand based filesharing system let me > > > know if you need any specific functionality... > > > > The obvious intermediate is to reinsert a small portion of a file. > > The normal case is (and will continue to be) that when a file becomes > > unretrievable, it's because one or more segments is only a couple > > blocks short of being retrievable. If you reinsert say 8 blocks out > > of each segment (1/32 of the file), you'll be reinserting on average 4 > > unretrievable blocks from each segment. That should be enough in a > > lot of cases. This is probably better than selective reinsert (the > > attacker doesn't get to choose which blocks you reinsert as easily), > > though it does mean reinserting more blocks (8 per segment when merely > > reinserting the correct 3 blocks might suffice). > > > > The simple defense against a mobile opennet attacker that has been > > proposed before would be particularly well suited to partial > > randomized reinserts. The insert comes with a time (randomized per > > block, to some time a bit before the reinsert started), and is only > > routed along connections that were established before that time, until > > it reaches some relatively low HTL (10?). This prevents the attacker > > from moving during the insert. On a large file that takes a long time > > to insert, this is problematic, because there aren't enough > > connections that are old enough to route along. For a partial > > reinsert, this is less of a concern, simply because it doesn't take as > > long. > > This is a very good point. What if we can improve on this further? > > By implementing long-term requests, we could have *all* the requests for a > splitfile go out *at once*, be routed immediately, and then return the data > over a long period. This means that: > 1) The data needs to be trickled back even if nodes go offline - either via > rerouting (but consider carefully how to make this safe, e.g. establishing > backup routes at the time, or using a node identifier for the next hop so we > can reroute via FOAFs without involving the originator so not giving a data > point away), or by waiting for the nodes to come back online. > 2) Load management needs to be able to deal with the fact that we have > thousands of requests in flight. This means it may not work on opennet, > because there is no underlying trust; although we could maybe have a > reputation system to build up some amount of trust. Trust can be translated > into capacity limits. > 3) The mobile attacker defence holds: If all the requests get routed inside a > few minutes, and then return data along fixed paths, the attacker has no > chance of moving towards the originator. And this works even for fairly big > files, without the overhead of tunnels, for requests and inserts of > predictable data. > 4) Overheads should be reasonable, because we can bundle a large number of > requests together efficiently. > 5) We get "burst" behaviour. If we have a fast connection, the data will be > returned fast. > 6) We get slow-return behaviour. In many cases it will take a long time for > the data to trickle back. At each hop it will make sense to send one key at a > time, if we happen to have multiple keys fully available. > 7) The node needs to be able to cope with a large number of requests pending: > We can keep the current code for routing them but once we have a route, the > requests as well as the transfers need to be threadless. > 8) We need to be able to specify that we want a fast response on a request > and that it should not get queued and trickled through/around offline nodes. > 9) We need a different way to handle timeouts. At the moment we detect when a > node is busted by not getting the data within X period. If the node has a > backlog of thousands of blocks then this is clearly not going to work. > However we do expect a response eventually. So we need some way to take this > into account and identify problems. > > This is an elaboration on a chain of thought that goes way back, thanks > particularly to the person who came up with the mobile attacker > countermeasure, but we've been pondering passive requests for a long time ... > It also combines ideas from CCN's hands off load management and various > people who have suggested load management changes over the years e.g. on > Frost... > > This looks like a major rewrite, maybe 0.10 era, but it would be worth it, > the potential benefits are rather enormous... > > Of course we'd still need tunnels for e.g. Frost posts, stuff where you have > multiple completed tasks over a period where the attacker could get a sample > from each one. > > PRINCIPLES IN DETAIL: > > 1. We send requests when we have requests. We queue them as little as > possible. Of course we have a queue of stuff we want to fetch - we recognise > blocks that we want - but in terms of the actual queues of requests to send, > we minimise this - stuff is sent immediately or near to immediately. > 2. We send all the requests for a single bundle at once. We queue other > bundles if necessary to fit into our capacity limit. If two bundles are > related (e.g. different levels of a splitfile or frost posts), we try to send > them close together. > 3. Any request in a bundle has a starting timestamp, which is the same or > very close for all the requests in the bundle. We don't route to any node > that has connected since the starting timestamp. > 4. Any given link (connection between a pair of nodes) has a capacity in > outstanding requests. On darknet this is fixed, or depends on the trust level > for the connection. On opennet this is fixed but much lower, or depends on > some sort of trust/history/reputation algorithm. > 5. Requests (and inserts) are routed reasonably quickly, but may be queued > briefly if necessary to achieve good routing. > 6. Once a request has reached its destination, where the data is available, > it is converted into a pending transfer. We exit the request thread and > handle it asynchronously. > 7. On any link there may be a large number of pending transfers, up to a > given capacity. If a request fails, we free up the capacity; hence we keep > the number of pending transfers under the limit. As long as we have data to > transfer, we transfer it. If we have a full blocks (e.g. if we are the data > source), we transfer that first, then look for another one, then look for > other blocks etc. > 8. As long as data is being transferred over a link, and requests are being > completed regularly (this is why we try to transfer one block at a time > rather than multiplexing everything), we don't timeout any of the transfers. > We may have to incorporate age into the error detection somehow. > 9. As a request reaches each hop, we give it the identifier for a node for an > alternative path. This can either be the previous hop or another node which > has agreed to be a fallback path, depending on security requirements. The > node identifier must be accessible within two hops of the node being routed > to. If the predecessor node goes offline, we will try to route to the > fallback path. If the fallback path is offline, we check whether we have the > persistent flag (which may be disallowed on opennet): If we have the > persistent flag we suspend the request until either the predecessor or the > fallback comes online, and write it to disk. If we don't have the persistent > flag, we store the returning data locally only, and penalise the predecessor > if it comes back online. Once we have sent a request we are committed to > accepting the returned data, this is essential to prevent probing-based > censorship attacks. This is why opennet is a problem with this scheme: If you > get any nontrivial trust from announcing, you can announce, connect to nodes, > use that capacity to DoS them and then disconnect and talk to another node. > However, if everyone has fast links, it could return data very quickly on > opennet... > 10. If our capacity limits are low then we will have to send requests (or > inserts) for big files in a series of separate bundles. We warn the user that > we cannot guarantee full security, and get a confirmation. Unless seclevel = > LOW, in which case we just proceed anyway without asking. One interesting > point is that this allows us to quantify the loss of security on opennet, > although arguably that will just result in more people getting connections to > random strangers out of band, many of whom will turn out to be NSA... > > Of course there are security risks with bursting, in that if an attacker has > access to traffic analysis as well as nodes close to the originator, he may > be able to tie the two together. But the above is not limited to bursting: > The principle is we burst the routing phase and then return the data as fast > or as slow as possible. It is compatible with sneakernet, it is compatible > with CBR links. > > Ideas? Challenges? Suggestions for how to deal with opennet in this framework? > One important security challenge is each time you request the same splitfile you are vulnerable. So either: 1) We route once, and reroute downstream. I.e. full passive requests. Which would combine very neatly with passive requests - when a peer comes online, check its bloom filters for anything that you or any of your pending requests want. However, this is a big step beyond the above, because it means freezing requests *that have not found a path to the data*. 2) We accept the security penalty and warn users before they enable poll-forever.
And of course, it is yet another reason to improve persistence radically. We have several options for that: - Chase up why triple inserts are *drastically* better at long term persistence, and probably modify inserts to capture this behaviour at a lower security and performance cost. - Implement bloom filter sharing. - Client layer persistence improvements.
signature.asc
Description: This is a digitally signed message part.
_______________________________________________ Devl mailing list Devl@freenetproject.org http://freenetproject.org/cgi-bin/mailman/listinfo/devl
