[freenet-dev] Fast *and* secure Freenet: Secure bursting and long term requests was Re: Planned changes to keys and UI
On Thursday 24 June 2010 18:28:42 Matthew Toseland wrote: > On Thursday 24 June 2010 04:05:48 Evan Daniel wrote: > > On Wed, Jun 23, 2010 at 5:43 PM, Matthew Toseland > > wrote: > > > On Wednesday 23 June 2010 20:33:50 Sich wrote: > > >> Le 23/06/2010 21:01, Matthew Toseland a ?crit : > > >> > Insert a random, safe key > > >> > This is much safer than the first option, but the key will be > > >> > different every time you or somebody else inserts the key. Use this if > > >> > you are the original source of some sensitive data. > > >> > > > >> > > > >> Very interesting for filesharing if we split the file. > > >> When some chunk are lost, you have only to reinsert those who are > > >> lost... But then we use much datastore... But it's more secure... > > >> Loosing datastore space is a big problem no ? > > > > > > If some people use the new key and some use the old then it's a problem. > > > If everyone uses one or the other it isn't. I guess this is another > > > reason to use par files etc (ugh). > > > > > > The next round of major changes (probably in 1255) will introduce > > > cross-segment redundancy which should improve the reliability of ?really > > > big files. > > > > > > Long term we may have selective reinsert support, but of course that > > > would be nearly as unsafe as reinserting the whole file to the same key > > > ... > > > > > > If you're building a reinsert-on-demand based filesharing system let me > > > know if you need any specific functionality... > > > > The obvious intermediate is to reinsert a small portion of a file. > > The normal case is (and will continue to be) that when a file becomes > > unretrievable, it's because one or more segments is only a couple > > blocks short of being retrievable. If you reinsert say 8 blocks out > > of each segment (1/32 of the file), you'll be reinserting on average 4 > > unretrievable blocks from each segment. That should be enough in a > > lot of cases. This is probably better than selective reinsert (the > > attacker doesn't get to choose which blocks you reinsert as easily), > > though it does mean reinserting more blocks (8 per segment when merely > > reinserting the correct 3 blocks might suffice). > > > > The simple defense against a mobile opennet attacker that has been > > proposed before would be particularly well suited to partial > > randomized reinserts. The insert comes with a time (randomized per > > block, to some time a bit before the reinsert started), and is only > > routed along connections that were established before that time, until > > it reaches some relatively low HTL (10?). This prevents the attacker > > from moving during the insert. On a large file that takes a long time > > to insert, this is problematic, because there aren't enough > > connections that are old enough to route along. For a partial > > reinsert, this is less of a concern, simply because it doesn't take as > > long. > > This is a very good point. What if we can improve on this further? > > By implementing long-term requests, we could have *all* the requests for a > splitfile go out *at once*, be routed immediately, and then return the data > over a long period. This means that: > 1) The data needs to be trickled back even if nodes go offline - either via > rerouting (but consider carefully how to make this safe, e.g. establishing > backup routes at the time, or using a node identifier for the next hop so we > can reroute via FOAFs without involving the originator so not giving a data > point away), or by waiting for the nodes to come back online. > 2) Load management needs to be able to deal with the fact that we have > thousands of requests in flight. This means it may not work on opennet, > because there is no underlying trust; although we could maybe have a > reputation system to build up some amount of trust. Trust can be translated > into capacity limits. > 3) The mobile attacker defence holds: If all the requests get routed inside a > few minutes, and then return data along fixed paths, the attacker has no > chance of moving towards the originator. And this works even for fairly big > files, without the overhead of tunnels, for requests and inserts of > predictable data. > 4) Overheads should be reasonable, because we can bundle a large number of > requests together efficiently. > 5) We get "burst" behaviour. If we have a fast connection, the data will be > returned fast. > 6) We get slow-return behaviour. In many cases it will take a long time for > the data to trickle back. At each hop it will make sense to send one key at a > time, if we happen to have multiple keys fully available. > 7) The node needs to be able to cope with a large number of requests pending: > We can keep the current code for routing them but once we have a route, the > requests as well as the transfers need to be threadless. > 8) We need to be able to specify that we want a fast response on a request > and that it should not get
[freenet-dev] Fast *and* secure Freenet: Secure bursting and long term requests was Re: Planned changes to keys and UI
On Thursday 24 June 2010 18:28:42 Matthew Toseland wrote: > On Thursday 24 June 2010 04:05:48 Evan Daniel wrote: > > On Wed, Jun 23, 2010 at 5:43 PM, Matthew Toseland > > wrote: > > > On Wednesday 23 June 2010 20:33:50 Sich wrote: > > >> Le 23/06/2010 21:01, Matthew Toseland a ?crit : > > >> > Insert a random, safe key > > >> > This is much safer than the first option, but the key will be > > >> > different every time you or somebody else inserts the key. Use this if > > >> > you are the original source of some sensitive data. > > >> > > > >> > > > >> Very interesting for filesharing if we split the file. > > >> When some chunk are lost, you have only to reinsert those who are > > >> lost... But then we use much datastore... But it's more secure... > > >> Loosing datastore space is a big problem no ? > > > > > > If some people use the new key and some use the old then it's a problem. > > > If everyone uses one or the other it isn't. I guess this is another > > > reason to use par files etc (ugh). > > > > > > The next round of major changes (probably in 1255) will introduce > > > cross-segment redundancy which should improve the reliability of ?really > > > big files. > > > > > > Long term we may have selective reinsert support, but of course that > > > would be nearly as unsafe as reinserting the whole file to the same key > > > ... > > > > > > If you're building a reinsert-on-demand based filesharing system let me > > > know if you need any specific functionality... > > > > The obvious intermediate is to reinsert a small portion of a file. > > The normal case is (and will continue to be) that when a file becomes > > unretrievable, it's because one or more segments is only a couple > > blocks short of being retrievable. If you reinsert say 8 blocks out > > of each segment (1/32 of the file), you'll be reinserting on average 4 > > unretrievable blocks from each segment. That should be enough in a > > lot of cases. This is probably better than selective reinsert (the > > attacker doesn't get to choose which blocks you reinsert as easily), > > though it does mean reinserting more blocks (8 per segment when merely > > reinserting the correct 3 blocks might suffice). > > > > The simple defense against a mobile opennet attacker that has been > > proposed before would be particularly well suited to partial > > randomized reinserts. The insert comes with a time (randomized per > > block, to some time a bit before the reinsert started), and is only > > routed along connections that were established before that time, until > > it reaches some relatively low HTL (10?). This prevents the attacker > > from moving during the insert. On a large file that takes a long time > > to insert, this is problematic, because there aren't enough > > connections that are old enough to route along. For a partial > > reinsert, this is less of a concern, simply because it doesn't take as > > long. > > This is a very good point. What if we can improve on this further? > > By implementing long-term requests, we could have *all* the requests for a > splitfile go out *at once*, be routed immediately, and then return the data > over a long period. This means that: > 1) The data needs to be trickled back even if nodes go offline - either via > rerouting (but consider carefully how to make this safe, e.g. establishing > backup routes at the time, or using a node identifier for the next hop so we > can reroute via FOAFs without involving the originator so not giving a data > point away), or by waiting for the nodes to come back online. > 2) Load management needs to be able to deal with the fact that we have > thousands of requests in flight. This means it may not work on opennet, > because there is no underlying trust; although we could maybe have a > reputation system to build up some amount of trust. Trust can be translated > into capacity limits. > 3) The mobile attacker defence holds: If all the requests get routed inside a > few minutes, and then return data along fixed paths, the attacker has no > chance of moving towards the originator. And this works even for fairly big > files, without the overhead of tunnels, for requests and inserts of > predictable data. > 4) Overheads should be reasonable, because we can bundle a large number of > requests together efficiently. > 5) We get "burst" behaviour. If we have a fast connection, the data will be > returned fast. > 6) We get slow-return behaviour. In many cases it will take a long time for > the data to trickle back. At each hop it will make sense to send one key at a > time, if we happen to have multiple keys fully available. > 7) The node needs to be able to cope with a large number of requests pending: > We can keep the current code for routing them but once we have a route, the > requests as well as the transfers need to be threadless. > 8) We need to be able to specify that we want a fast response on a request > and that it should not get
[freenet-dev] Fast *and* secure Freenet: Secure bursting and long term requests was Re: Planned changes to keys and UI
On Thursday 24 June 2010 18:28:42 Matthew Toseland wrote: > On Thursday 24 June 2010 04:05:48 Evan Daniel wrote: > > On Wed, Jun 23, 2010 at 5:43 PM, Matthew Toseland > > wrote: > > > On Wednesday 23 June 2010 20:33:50 Sich wrote: > > >> Le 23/06/2010 21:01, Matthew Toseland a ?crit : > > >> > Insert a random, safe key > > >> > This is much safer than the first option, but the key will be > > >> > different every time you or somebody else inserts the key. Use this if > > >> > you are the original source of some sensitive data. > > >> > > > >> > > > >> Very interesting for filesharing if we split the file. > > >> When some chunk are lost, you have only to reinsert those who are > > >> lost... But then we use much datastore... But it's more secure... > > >> Loosing datastore space is a big problem no ? > > > > > > If some people use the new key and some use the old then it's a problem. > > > If everyone uses one or the other it isn't. I guess this is another > > > reason to use par files etc (ugh). > > > > > > The next round of major changes (probably in 1255) will introduce > > > cross-segment redundancy which should improve the reliability of ?really > > > big files. > > > > > > Long term we may have selective reinsert support, but of course that > > > would be nearly as unsafe as reinserting the whole file to the same key > > > ... > > > > > > If you're building a reinsert-on-demand based filesharing system let me > > > know if you need any specific functionality... > > > > The obvious intermediate is to reinsert a small portion of a file. > > The normal case is (and will continue to be) that when a file becomes > > unretrievable, it's because one or more segments is only a couple > > blocks short of being retrievable. If you reinsert say 8 blocks out > > of each segment (1/32 of the file), you'll be reinserting on average 4 > > unretrievable blocks from each segment. That should be enough in a > > lot of cases. This is probably better than selective reinsert (the > > attacker doesn't get to choose which blocks you reinsert as easily), > > though it does mean reinserting more blocks (8 per segment when merely > > reinserting the correct 3 blocks might suffice). > > > > The simple defense against a mobile opennet attacker that has been > > proposed before would be particularly well suited to partial > > randomized reinserts. The insert comes with a time (randomized per > > block, to some time a bit before the reinsert started), and is only > > routed along connections that were established before that time, until > > it reaches some relatively low HTL (10?). This prevents the attacker > > from moving during the insert. On a large file that takes a long time > > to insert, this is problematic, because there aren't enough > > connections that are old enough to route along. For a partial > > reinsert, this is less of a concern, simply because it doesn't take as > > long. > > This is a very good point. What if we can improve on this further? > > By implementing long-term requests, we could have *all* the requests for a > splitfile go out *at once*, be routed immediately, and then return the data > over a long period. This means that: > 1) The data needs to be trickled back even if nodes go offline - either via > rerouting (but consider carefully how to make this safe, e.g. establishing > backup routes at the time, or using a node identifier for the next hop so we > can reroute via FOAFs without involving the originator so not giving a data > point away), or by waiting for the nodes to come back online. > 2) Load management needs to be able to deal with the fact that we have > thousands of requests in flight. This means it may not work on opennet, > because there is no underlying trust; although we could maybe have a > reputation system to build up some amount of trust. Trust can be translated > into capacity limits. > 3) The mobile attacker defence holds: If all the requests get routed inside a > few minutes, and then return data along fixed paths, the attacker has no > chance of moving towards the originator. And this works even for fairly big > files, without the overhead of tunnels, for requests and inserts of > predictable data. > 4) Overheads should be reasonable, because we can bundle a large number of > requests together efficiently. > 5) We get "burst" behaviour. If we have a fast connection, the data will be > returned fast. > 6) We get slow-return behaviour. In many cases it will take a long time for > the data to trickle back. At each hop it will make sense to send one key at a > time, if we happen to have multiple keys fully available. > 7) The node needs to be able to cope with a large number of requests pending: > We can keep the current code for routing them but once we have a route, the > requests as well as the transfers need to be threadless. > 8) We need to be able to specify that we want a fast response on a request > and that it should not get
[freenet-dev] Fast *and* secure Freenet: Secure bursting and long term requests was Re: Planned changes to keys and UI
On Thursday 24 June 2010 04:05:48 Evan Daniel wrote: > On Wed, Jun 23, 2010 at 5:43 PM, Matthew Toseland > wrote: > > On Wednesday 23 June 2010 20:33:50 Sich wrote: > >> Le 23/06/2010 21:01, Matthew Toseland a ?crit : > >> > Insert a random, safe key > >> > This is much safer than the first option, but the key will be different > >> > every time you or somebody else inserts the key. Use this if you are the > >> > original source of some sensitive data. > >> > > >> > > >> Very interesting for filesharing if we split the file. > >> When some chunk are lost, you have only to reinsert those who are > >> lost... But then we use much datastore... But it's more secure... > >> Loosing datastore space is a big problem no ? > > > > If some people use the new key and some use the old then it's a problem. If > > everyone uses one or the other it isn't. I guess this is another reason to > > use par files etc (ugh). > > > > The next round of major changes (probably in 1255) will introduce > > cross-segment redundancy which should improve the reliability of ?really > > big files. > > > > Long term we may have selective reinsert support, but of course that would > > be nearly as unsafe as reinserting the whole file to the same key ... > > > > If you're building a reinsert-on-demand based filesharing system let me > > know if you need any specific functionality... > > The obvious intermediate is to reinsert a small portion of a file. > The normal case is (and will continue to be) that when a file becomes > unretrievable, it's because one or more segments is only a couple > blocks short of being retrievable. If you reinsert say 8 blocks out > of each segment (1/32 of the file), you'll be reinserting on average 4 > unretrievable blocks from each segment. That should be enough in a > lot of cases. This is probably better than selective reinsert (the > attacker doesn't get to choose which blocks you reinsert as easily), > though it does mean reinserting more blocks (8 per segment when merely > reinserting the correct 3 blocks might suffice). > > The simple defense against a mobile opennet attacker that has been > proposed before would be particularly well suited to partial > randomized reinserts. The insert comes with a time (randomized per > block, to some time a bit before the reinsert started), and is only > routed along connections that were established before that time, until > it reaches some relatively low HTL (10?). This prevents the attacker > from moving during the insert. On a large file that takes a long time > to insert, this is problematic, because there aren't enough > connections that are old enough to route along. For a partial > reinsert, this is less of a concern, simply because it doesn't take as > long. This is a very good point. What if we can improve on this further? By implementing long-term requests, we could have *all* the requests for a splitfile go out *at once*, be routed immediately, and then return the data over a long period. This means that: 1) The data needs to be trickled back even if nodes go offline - either via rerouting (but consider carefully how to make this safe, e.g. establishing backup routes at the time, or using a node identifier for the next hop so we can reroute via FOAFs without involving the originator so not giving a data point away), or by waiting for the nodes to come back online. 2) Load management needs to be able to deal with the fact that we have thousands of requests in flight. This means it may not work on opennet, because there is no underlying trust; although we could maybe have a reputation system to build up some amount of trust. Trust can be translated into capacity limits. 3) The mobile attacker defence holds: If all the requests get routed inside a few minutes, and then return data along fixed paths, the attacker has no chance of moving towards the originator. And this works even for fairly big files, without the overhead of tunnels, for requests and inserts of predictable data. 4) Overheads should be reasonable, because we can bundle a large number of requests together efficiently. 5) We get "burst" behaviour. If we have a fast connection, the data will be returned fast. 6) We get slow-return behaviour. In many cases it will take a long time for the data to trickle back. At each hop it will make sense to send one key at a time, if we happen to have multiple keys fully available. 7) The node needs to be able to cope with a large number of requests pending: We can keep the current code for routing them but once we have a route, the requests as well as the transfers need to be threadless. 8) We need to be able to specify that we want a fast response on a request and that it should not get queued and trickled through/around offline nodes. 9) We need a different way to handle timeouts. At the moment we detect when a node is busted by not getting the data within X period. If the node has a backlog of thousands of blocks then
[freenet-dev] Fast *and* secure Freenet: Secure bursting and long term requests was Re: Planned changes to keys and UI
On Thursday 24 June 2010 04:05:48 Evan Daniel wrote: On Wed, Jun 23, 2010 at 5:43 PM, Matthew Toseland t...@amphibian.dyndns.org wrote: On Wednesday 23 June 2010 20:33:50 Sich wrote: Le 23/06/2010 21:01, Matthew Toseland a écrit : Insert a random, safe key This is much safer than the first option, but the key will be different every time you or somebody else inserts the key. Use this if you are the original source of some sensitive data. Very interesting for filesharing if we split the file. When some chunk are lost, you have only to reinsert those who are lost... But then we use much datastore... But it's more secure... Loosing datastore space is a big problem no ? If some people use the new key and some use the old then it's a problem. If everyone uses one or the other it isn't. I guess this is another reason to use par files etc (ugh). The next round of major changes (probably in 1255) will introduce cross-segment redundancy which should improve the reliability of really big files. Long term we may have selective reinsert support, but of course that would be nearly as unsafe as reinserting the whole file to the same key ... If you're building a reinsert-on-demand based filesharing system let me know if you need any specific functionality... The obvious intermediate is to reinsert a small portion of a file. The normal case is (and will continue to be) that when a file becomes unretrievable, it's because one or more segments is only a couple blocks short of being retrievable. If you reinsert say 8 blocks out of each segment (1/32 of the file), you'll be reinserting on average 4 unretrievable blocks from each segment. That should be enough in a lot of cases. This is probably better than selective reinsert (the attacker doesn't get to choose which blocks you reinsert as easily), though it does mean reinserting more blocks (8 per segment when merely reinserting the correct 3 blocks might suffice). The simple defense against a mobile opennet attacker that has been proposed before would be particularly well suited to partial randomized reinserts. The insert comes with a time (randomized per block, to some time a bit before the reinsert started), and is only routed along connections that were established before that time, until it reaches some relatively low HTL (10?). This prevents the attacker from moving during the insert. On a large file that takes a long time to insert, this is problematic, because there aren't enough connections that are old enough to route along. For a partial reinsert, this is less of a concern, simply because it doesn't take as long. This is a very good point. What if we can improve on this further? By implementing long-term requests, we could have *all* the requests for a splitfile go out *at once*, be routed immediately, and then return the data over a long period. This means that: 1) The data needs to be trickled back even if nodes go offline - either via rerouting (but consider carefully how to make this safe, e.g. establishing backup routes at the time, or using a node identifier for the next hop so we can reroute via FOAFs without involving the originator so not giving a data point away), or by waiting for the nodes to come back online. 2) Load management needs to be able to deal with the fact that we have thousands of requests in flight. This means it may not work on opennet, because there is no underlying trust; although we could maybe have a reputation system to build up some amount of trust. Trust can be translated into capacity limits. 3) The mobile attacker defence holds: If all the requests get routed inside a few minutes, and then return data along fixed paths, the attacker has no chance of moving towards the originator. And this works even for fairly big files, without the overhead of tunnels, for requests and inserts of predictable data. 4) Overheads should be reasonable, because we can bundle a large number of requests together efficiently. 5) We get burst behaviour. If we have a fast connection, the data will be returned fast. 6) We get slow-return behaviour. In many cases it will take a long time for the data to trickle back. At each hop it will make sense to send one key at a time, if we happen to have multiple keys fully available. 7) The node needs to be able to cope with a large number of requests pending: We can keep the current code for routing them but once we have a route, the requests as well as the transfers need to be threadless. 8) We need to be able to specify that we want a fast response on a request and that it should not get queued and trickled through/around offline nodes. 9) We need a different way to handle timeouts. At the moment we detect when a node is busted by not getting the data within X period. If the node has a backlog of thousands of blocks then this is clearly not going to work. However we do expect a response
Re: [freenet-dev] Fast *and* secure Freenet: Secure bursting and long term requests was Re: Planned changes to keys and UI
On Thursday 24 June 2010 18:28:42 Matthew Toseland wrote: On Thursday 24 June 2010 04:05:48 Evan Daniel wrote: On Wed, Jun 23, 2010 at 5:43 PM, Matthew Toseland t...@amphibian.dyndns.org wrote: On Wednesday 23 June 2010 20:33:50 Sich wrote: Le 23/06/2010 21:01, Matthew Toseland a écrit : Insert a random, safe key This is much safer than the first option, but the key will be different every time you or somebody else inserts the key. Use this if you are the original source of some sensitive data. Very interesting for filesharing if we split the file. When some chunk are lost, you have only to reinsert those who are lost... But then we use much datastore... But it's more secure... Loosing datastore space is a big problem no ? If some people use the new key and some use the old then it's a problem. If everyone uses one or the other it isn't. I guess this is another reason to use par files etc (ugh). The next round of major changes (probably in 1255) will introduce cross-segment redundancy which should improve the reliability of really big files. Long term we may have selective reinsert support, but of course that would be nearly as unsafe as reinserting the whole file to the same key ... If you're building a reinsert-on-demand based filesharing system let me know if you need any specific functionality... The obvious intermediate is to reinsert a small portion of a file. The normal case is (and will continue to be) that when a file becomes unretrievable, it's because one or more segments is only a couple blocks short of being retrievable. If you reinsert say 8 blocks out of each segment (1/32 of the file), you'll be reinserting on average 4 unretrievable blocks from each segment. That should be enough in a lot of cases. This is probably better than selective reinsert (the attacker doesn't get to choose which blocks you reinsert as easily), though it does mean reinserting more blocks (8 per segment when merely reinserting the correct 3 blocks might suffice). The simple defense against a mobile opennet attacker that has been proposed before would be particularly well suited to partial randomized reinserts. The insert comes with a time (randomized per block, to some time a bit before the reinsert started), and is only routed along connections that were established before that time, until it reaches some relatively low HTL (10?). This prevents the attacker from moving during the insert. On a large file that takes a long time to insert, this is problematic, because there aren't enough connections that are old enough to route along. For a partial reinsert, this is less of a concern, simply because it doesn't take as long. This is a very good point. What if we can improve on this further? By implementing long-term requests, we could have *all* the requests for a splitfile go out *at once*, be routed immediately, and then return the data over a long period. This means that: 1) The data needs to be trickled back even if nodes go offline - either via rerouting (but consider carefully how to make this safe, e.g. establishing backup routes at the time, or using a node identifier for the next hop so we can reroute via FOAFs without involving the originator so not giving a data point away), or by waiting for the nodes to come back online. 2) Load management needs to be able to deal with the fact that we have thousands of requests in flight. This means it may not work on opennet, because there is no underlying trust; although we could maybe have a reputation system to build up some amount of trust. Trust can be translated into capacity limits. 3) The mobile attacker defence holds: If all the requests get routed inside a few minutes, and then return data along fixed paths, the attacker has no chance of moving towards the originator. And this works even for fairly big files, without the overhead of tunnels, for requests and inserts of predictable data. 4) Overheads should be reasonable, because we can bundle a large number of requests together efficiently. 5) We get burst behaviour. If we have a fast connection, the data will be returned fast. 6) We get slow-return behaviour. In many cases it will take a long time for the data to trickle back. At each hop it will make sense to send one key at a time, if we happen to have multiple keys fully available. 7) The node needs to be able to cope with a large number of requests pending: We can keep the current code for routing them but once we have a route, the requests as well as the transfers need to be threadless. 8) We need to be able to specify that we want a fast response on a request and that it should not get queued and trickled through/around offline nodes. 9) We need a different way to handle timeouts. At the moment we detect when a node is busted by not getting
Re: [freenet-dev] Fast *and* secure Freenet: Secure bursting and long term requests was Re: Planned changes to keys and UI
On Thursday 24 June 2010 18:28:42 Matthew Toseland wrote: On Thursday 24 June 2010 04:05:48 Evan Daniel wrote: On Wed, Jun 23, 2010 at 5:43 PM, Matthew Toseland t...@amphibian.dyndns.org wrote: On Wednesday 23 June 2010 20:33:50 Sich wrote: Le 23/06/2010 21:01, Matthew Toseland a écrit : Insert a random, safe key This is much safer than the first option, but the key will be different every time you or somebody else inserts the key. Use this if you are the original source of some sensitive data. Very interesting for filesharing if we split the file. When some chunk are lost, you have only to reinsert those who are lost... But then we use much datastore... But it's more secure... Loosing datastore space is a big problem no ? If some people use the new key and some use the old then it's a problem. If everyone uses one or the other it isn't. I guess this is another reason to use par files etc (ugh). The next round of major changes (probably in 1255) will introduce cross-segment redundancy which should improve the reliability of really big files. Long term we may have selective reinsert support, but of course that would be nearly as unsafe as reinserting the whole file to the same key ... If you're building a reinsert-on-demand based filesharing system let me know if you need any specific functionality... The obvious intermediate is to reinsert a small portion of a file. The normal case is (and will continue to be) that when a file becomes unretrievable, it's because one or more segments is only a couple blocks short of being retrievable. If you reinsert say 8 blocks out of each segment (1/32 of the file), you'll be reinserting on average 4 unretrievable blocks from each segment. That should be enough in a lot of cases. This is probably better than selective reinsert (the attacker doesn't get to choose which blocks you reinsert as easily), though it does mean reinserting more blocks (8 per segment when merely reinserting the correct 3 blocks might suffice). The simple defense against a mobile opennet attacker that has been proposed before would be particularly well suited to partial randomized reinserts. The insert comes with a time (randomized per block, to some time a bit before the reinsert started), and is only routed along connections that were established before that time, until it reaches some relatively low HTL (10?). This prevents the attacker from moving during the insert. On a large file that takes a long time to insert, this is problematic, because there aren't enough connections that are old enough to route along. For a partial reinsert, this is less of a concern, simply because it doesn't take as long. This is a very good point. What if we can improve on this further? By implementing long-term requests, we could have *all* the requests for a splitfile go out *at once*, be routed immediately, and then return the data over a long period. This means that: 1) The data needs to be trickled back even if nodes go offline - either via rerouting (but consider carefully how to make this safe, e.g. establishing backup routes at the time, or using a node identifier for the next hop so we can reroute via FOAFs without involving the originator so not giving a data point away), or by waiting for the nodes to come back online. 2) Load management needs to be able to deal with the fact that we have thousands of requests in flight. This means it may not work on opennet, because there is no underlying trust; although we could maybe have a reputation system to build up some amount of trust. Trust can be translated into capacity limits. 3) The mobile attacker defence holds: If all the requests get routed inside a few minutes, and then return data along fixed paths, the attacker has no chance of moving towards the originator. And this works even for fairly big files, without the overhead of tunnels, for requests and inserts of predictable data. 4) Overheads should be reasonable, because we can bundle a large number of requests together efficiently. 5) We get burst behaviour. If we have a fast connection, the data will be returned fast. 6) We get slow-return behaviour. In many cases it will take a long time for the data to trickle back. At each hop it will make sense to send one key at a time, if we happen to have multiple keys fully available. 7) The node needs to be able to cope with a large number of requests pending: We can keep the current code for routing them but once we have a route, the requests as well as the transfers need to be threadless. 8) We need to be able to specify that we want a fast response on a request and that it should not get queued and trickled through/around offline nodes. 9) We need a different way to handle timeouts. At the moment we detect when a node is busted by not getting
Re: [freenet-dev] Fast *and* secure Freenet: Secure bursting and long term requests was Re: Planned changes to keys and UI
On Thursday 24 June 2010 18:28:42 Matthew Toseland wrote: On Thursday 24 June 2010 04:05:48 Evan Daniel wrote: On Wed, Jun 23, 2010 at 5:43 PM, Matthew Toseland t...@amphibian.dyndns.org wrote: On Wednesday 23 June 2010 20:33:50 Sich wrote: Le 23/06/2010 21:01, Matthew Toseland a écrit : Insert a random, safe key This is much safer than the first option, but the key will be different every time you or somebody else inserts the key. Use this if you are the original source of some sensitive data. Very interesting for filesharing if we split the file. When some chunk are lost, you have only to reinsert those who are lost... But then we use much datastore... But it's more secure... Loosing datastore space is a big problem no ? If some people use the new key and some use the old then it's a problem. If everyone uses one or the other it isn't. I guess this is another reason to use par files etc (ugh). The next round of major changes (probably in 1255) will introduce cross-segment redundancy which should improve the reliability of really big files. Long term we may have selective reinsert support, but of course that would be nearly as unsafe as reinserting the whole file to the same key ... If you're building a reinsert-on-demand based filesharing system let me know if you need any specific functionality... The obvious intermediate is to reinsert a small portion of a file. The normal case is (and will continue to be) that when a file becomes unretrievable, it's because one or more segments is only a couple blocks short of being retrievable. If you reinsert say 8 blocks out of each segment (1/32 of the file), you'll be reinserting on average 4 unretrievable blocks from each segment. That should be enough in a lot of cases. This is probably better than selective reinsert (the attacker doesn't get to choose which blocks you reinsert as easily), though it does mean reinserting more blocks (8 per segment when merely reinserting the correct 3 blocks might suffice). The simple defense against a mobile opennet attacker that has been proposed before would be particularly well suited to partial randomized reinserts. The insert comes with a time (randomized per block, to some time a bit before the reinsert started), and is only routed along connections that were established before that time, until it reaches some relatively low HTL (10?). This prevents the attacker from moving during the insert. On a large file that takes a long time to insert, this is problematic, because there aren't enough connections that are old enough to route along. For a partial reinsert, this is less of a concern, simply because it doesn't take as long. This is a very good point. What if we can improve on this further? By implementing long-term requests, we could have *all* the requests for a splitfile go out *at once*, be routed immediately, and then return the data over a long period. This means that: 1) The data needs to be trickled back even if nodes go offline - either via rerouting (but consider carefully how to make this safe, e.g. establishing backup routes at the time, or using a node identifier for the next hop so we can reroute via FOAFs without involving the originator so not giving a data point away), or by waiting for the nodes to come back online. 2) Load management needs to be able to deal with the fact that we have thousands of requests in flight. This means it may not work on opennet, because there is no underlying trust; although we could maybe have a reputation system to build up some amount of trust. Trust can be translated into capacity limits. 3) The mobile attacker defence holds: If all the requests get routed inside a few minutes, and then return data along fixed paths, the attacker has no chance of moving towards the originator. And this works even for fairly big files, without the overhead of tunnels, for requests and inserts of predictable data. 4) Overheads should be reasonable, because we can bundle a large number of requests together efficiently. 5) We get burst behaviour. If we have a fast connection, the data will be returned fast. 6) We get slow-return behaviour. In many cases it will take a long time for the data to trickle back. At each hop it will make sense to send one key at a time, if we happen to have multiple keys fully available. 7) The node needs to be able to cope with a large number of requests pending: We can keep the current code for routing them but once we have a route, the requests as well as the transfers need to be threadless. 8) We need to be able to specify that we want a fast response on a request and that it should not get queued and trickled through/around offline nodes. 9) We need a different way to handle timeouts. At the moment we detect when a node is busted by not getting
