Author: toad Date: 2005-10-21 13:17:13 +0000 (Fri, 21 Oct 2005) New Revision: 7441
Added: trunk/freenet/devnotes/pubsub/emails.txt Modified: trunk/freenet/devnotes/pubsub/linyos-on-pubsub.txt Log: More pubsub docs. Added: trunk/freenet/devnotes/pubsub/emails.txt =================================================================== --- trunk/freenet/devnotes/pubsub/emails.txt 2005-10-21 13:07:28 UTC (rev 7440) +++ trunk/freenet/devnotes/pubsub/emails.txt 2005-10-21 13:17:13 UTC (rev 7441) @@ -0,0 +1,572 @@ +To: [email protected] +From: Matthew Toseland <[EMAIL PROTECTED]> +Cc: [EMAIL PROTECTED] +Subject: [Tech] Publish/subscribe redux +X-BeenThere: [email protected] +X-Mailman-Version: 2.1.5 +Reply-To: [email protected] +List-Id: tech.freenetproject.org +List-Unsubscribe: <http://emu.freenetproject.org/cgi-bin/mailman/listinfo/tech>, + <mailto:[EMAIL PROTECTED]> +List-Archive: <http://emu.freenetproject.org/pipermail/tech> +List-Post: <mailto:[email protected]> +List-Help: <mailto:[EMAIL PROTECTED]> +List-Subscribe: <http://emu.freenetproject.org/cgi-bin/mailman/listinfo/tech>, + <mailto:[EMAIL PROTECTED]> + +<snip signature> + +[21:02] <toad_> high bandwidth streams => vulnerable to bottlenecks => +striping, ideally use passive requests +[21:02] <toad_> medium bandwidth, low latency streams => passive +requests are bad, preferable to use a single path +[21:03] <toad_> low bandwidth streams => either way is equally +reasonable +[21:03] <toad_> given that we won't be implementing 1:1 streams for some +time, i don't know how important the middle case is + +linyos is concerned about bottlenecks. If we channel everything through +one key, then that makes a bottleneck - if the node is slow, or if any +node on the insert chain is slow, then it is a problem. + +Conclusions: +- It is likely that the current approach of imposing arbitrary limits + will not work. We need to have the actual data in the datastore, with + LRU. This means, ultimately, implementing pub/sub over passive + requests - there are two approaches: + a) Spread spectrum - we subscribe to [EMAIL PROTECTED]/1, [EMAIL PROTECTED]/2, /3 etc. + This is more or less immune to "bottleneck" effects. But it is not + very efficient. + b) TUK-based - we send a passive request for a TUK, with a flag to + indicate that the passive request not be deleted when it fires. It + will then fire when new data comes in. This is more efficient than + the above because we do not need to maintain a large pool of + passive requests for future keys. But it is vulnerable to + bottleneck effects. +- We have not promised anyone that we will implement TUKs for 0.7.0, and + we should not do so unless it is necessary. +- If the current code on pub/sub is useful, we should get it working for + 0.7.0. IMHO it isn't directly, but the thinking that went into it is. + That can be reused at a later date. +- It is possible to implement both fast and slow streams over passive + requests and SSKs. +- Passive requests have many of the same issues that publish/subscribe + has. There are however simple (but crude) solutions. The obvious one + being if we lose our parent node, instead of rerouting we just + forcibly disconnect all our dependants. They will then retry. + Coalescing can work exactly as in regular requests to ensure that the + passive-request tree does not get into loops. + +Therefore: +- We should shelve the whole publish/subscribe project for the time + being. +- We should implement the basic functionality of freenet - splitfiles, + SSKs, FCPv2, etc. +- Passive requests, possibly TUKs, publish/subscribe streams etc. should + go into a future release. IMHO this should be 0.8.0. Premix routing + and 1:1 streams (in cooperation with I2P or otherwise) should go into + 0.9.0. :) +-- +Matthew J Toseland - [EMAIL PROTECTED] +Freenet Project Official Codemonkey - http://freenetproject.org/ +ICTHUS - Nothing is impossible. Our Boss says so. + +------------------------------------------------------------------------- +PREVIOUSLY +------------------------------------------------------------------------- + +To: [email protected] +From: Matthew Toseland <[EMAIL PROTECTED]> +Subject: [Tech] Pub/sub = passive requests + TUKs +X-BeenThere: [email protected] +X-Mailman-Version: 2.1.5 +Reply-To: [email protected] +List-Id: tech.freenetproject.org +List-Unsubscribe: <http://emu.freenetproject.org/cgi-bin/mailman/listinfo/tech>, + <mailto:[EMAIL PROTECTED]> +List-Archive: <http://emu.freenetproject.org/pipermail/tech> +List-Post: <mailto:[email protected]> +List-Help: <mailto:[EMAIL PROTECTED]> +List-Subscribe: <http://emu.freenetproject.org/cgi-bin/mailman/listinfo/tech>, + <mailto:[EMAIL PROTECTED]> + +Suppose: +- We have a separate store for SSKs. They have 1kB of payload, plus the + headers. +- SSKs are versioned, with a timestamp. I.e. they become TUKs. Requests + can optionally propagate far enough to fetch all values. There can be + any number of SSKs with the same key and different version numbers / + datestamps. Requests can ask for either a single specific version, the + most recent version, or everything since a given revision. Only the + data blocks which are actually returned are promoted in the cache. +- To subscribe to a stream, I send a request out for the key I want, + with the minimum acceptable version number/datestamp, plus three flags: + - Passive request - If the data is not found, keep a tag on each node + on the chain, and if it ever passes through any of the nodes on the + chain, pass it back to me. + - TUK - Return the most recent version of the SSK. + - Subscribe - Requires the other two. Means the tag persists. + Obviously it will be deleted if the originator rejects the data, or + unsubscribes. + +Now, how would this actually work? + +Each node we are connected to can have up to 1000 subscriptions. Each +subscription is for a key - this can be an SSK or a CHK - and if it is +an SSK, it can have the "persist" flag. If it does not have this flag, +or this flag is false, then when the data is found, the subscription is +removed. If we no do not have enough subscription quota left for a given +subscription, for that node, or for ourselves with another node, we +reject the request attempting to add the subscription with a +RejectOverload. Nodes can unsubscribe. Each subscription can have an +expiry date (actually, perhaps we should impose one). + +Passive requests can only be created by actual requests with the passive +request flag. The request will be routed as normal, until HTL runs out +(or it finds the data if the TUK flag is not enabled). + +Now, how do we make this scale?: +1. If the TUK flag is off, and the request reaches a node which has the +data, we return it, and the request ends. No passive requests are +installed. +2. If any request for a TUK, even if not passive, reaches a node which +already has a valid subscription at a higher or equal HTL for the key, +then the request ends at that point, and returns the data that node has. +If the passive-request flag is enabled, then passive request tags are +added up to that node, and that node adds one for the node connecting to +it if necessary. +3. What to do about looping and coalescing? One option is to not +coalesce at all with running requests, and either forward the request +as-is or reject it... more in my next email... +-- +Matthew J Toseland - [EMAIL PROTECTED] +Freenet Project Official Codemonkey - http://freenetproject.org/ +ICTHUS - Nothing is impossible. Our Boss says so. + + +To: [email protected] +Subject: Re: [Tech] Pub/sub = passive requests + TUKs +From: Matthew Toseland <[EMAIL PROTECTED]> +X-BeenThere: [email protected] +X-Mailman-Version: 2.1.5 +Reply-To: [email protected] +List-Id: tech.freenetproject.org +List-Unsubscribe: <http://emu.freenetproject.org/cgi-bin/mailman/listinfo/tech>, + <mailto:[EMAIL PROTECTED]> +List-Archive: <http://emu.freenetproject.org/pipermail/tech> +List-Post: <mailto:[email protected]> +List-Help: <mailto:[EMAIL PROTECTED]> +List-Subscribe: <http://emu.freenetproject.org/cgi-bin/mailman/listinfo/tech>, + <mailto:[EMAIL PROTECTED]> + +On Thu, Oct 20, 2005 at 06:54:42PM +0100, Matthew Toseland wrote: +> Suppose: +> - We have a separate store for SSKs. They have 1kB of payload, plus the +> headers. +> - SSKs are versioned, with a timestamp. I.e. they become TUKs. Requests +> can optionally propagate far enough to fetch all values. There can be +> any number of SSKs with the same key and different version numbers / +> datestamps. Requests can ask for either a single specific version, the +> most recent version, or everything since a given revision. Only the +> data blocks which are actually returned are promoted in the cache. +> - To subscribe to a stream, I send a request out for the key I want, +> with the minimum acceptable version number/datestamp, plus three flags: +> - Passive request - If the data is not found, keep a tag on each node +> on the chain, and if it ever passes through any of the nodes on the +> chain, pass it back to me. +> - TUK - Return the most recent version of the SSK. +> - Subscribe - Requires the other two. Means the tag persists. +> Obviously it will be deleted if the originator rejects the data, or +> unsubscribes. +> +> Now, how would this actually work? +> +> Each node we are connected to can have up to 1000 subscriptions. Each +> subscription is for a key - this can be an SSK or a CHK - and if it is +> an SSK, it can have the "persist" flag. If it does not have this flag, +> or this flag is false, then when the data is found, the subscription is +> removed. If we no do not have enough subscription quota left for a given +> subscription, for that node, or for ourselves with another node, we +> reject the request attempting to add the subscription with a +> RejectOverload. Nodes can unsubscribe. Each subscription can have an +> expiry date (actually, perhaps we should impose one). +> +> Passive requests can only be created by actual requests with the passive +> request flag. The request will be routed as normal, until HTL runs out +> (or it finds the data if the TUK flag is not enabled). +> +> Now, how do we make this scale?: +> 1. If the TUK flag is off, and the request reaches a node which has the +> data, we return it, and the request ends. No passive requests are +> installed. +> 2. If any request for a TUK, even if not passive, reaches a node which +> already has a valid subscription at a higher or equal HTL for the key, +> then the request ends at that point, and returns the data that node has. +> If the passive-request flag is enabled, then passive request tags are +> added up to that node, and that node adds one for the node connecting to +> it if necessary. +> 3. What to do about looping and coalescing? One option is to not +> coalesce at all with running requests, and either forward the request +> as-is or reject it... more in my next email... + +As far as looping and coalescing goes: We can coalesce the requests +exactly as proposed for regular requests. So the request will not get +stuck by eating its own tail. As long as the network isn't totally +degenerate this shouldn't be a big problem. Also, if we do hit our own +tail, we could have a "lightweight subscribe" which isn't refcounted, +but passes packets both ways to strengthen the network. + +As far as resubscription and renewals go, there are two options really: +1. Try to get the network to maintain the subscriptions itself; +automatic resubscription etc. +2. Make the client resubscribe itself every so often. We would have to +let some of their requests through... +3. Don't bother. Switch streams from time to time at the client level. +The problem with this is that we will still have to deal with situations +such as the parent of a node being disconnected. + + +-- +Matthew J Toseland - [EMAIL PROTECTED] +Freenet Project Official Codemonkey - http://freenetproject.org/ +ICTHUS - Nothing is impossible. Our Boss says so. + + +To: [email protected] +Subject: Re: [Tech] Pub/sub = passive requests + TUKs +From: Matthew Toseland <[EMAIL PROTECTED]> +X-BeenThere: [email protected] +X-Mailman-Version: 2.1.5 +Reply-To: [email protected] +List-Id: tech.freenetproject.org +List-Unsubscribe: <http://emu.freenetproject.org/cgi-bin/mailman/listinfo/tech>, + <mailto:[EMAIL PROTECTED]> +List-Archive: <http://emu.freenetproject.org/pipermail/tech> +List-Post: <mailto:[email protected]> +List-Help: <mailto:[EMAIL PROTECTED]> +List-Subscribe: <http://emu.freenetproject.org/cgi-bin/mailman/listinfo/tech>, + <mailto:[EMAIL PROTECTED]> + +On Thu, Oct 20, 2005 at 07:46:46PM +0100, Matthew Toseland wrote: +> On Thu, Oct 20, 2005 at 06:54:42PM +0100, Matthew Toseland wrote: +> > Suppose: +> > - We have a separate store for SSKs. They have 1kB of payload, plus the +> > headers. +> > - SSKs are versioned, with a timestamp. I.e. they become TUKs. Requests +> > can optionally propagate far enough to fetch all values. There can be +> > any number of SSKs with the same key and different version numbers / +> > datestamps. Requests can ask for either a single specific version, the +> > most recent version, or everything since a given revision. Only the +> > data blocks which are actually returned are promoted in the cache. +> > - To subscribe to a stream, I send a request out for the key I want, +> > with the minimum acceptable version number/datestamp, plus three flags: +> > - Passive request - If the data is not found, keep a tag on each node +> > on the chain, and if it ever passes through any of the nodes on the +> > chain, pass it back to me. +> > - TUK - Return the most recent version of the SSK. +> > - Subscribe - Requires the other two. Means the tag persists. +> > Obviously it will be deleted if the originator rejects the data, or +> > unsubscribes. +> > +> > Now, how would this actually work? +> > +> > Each node we are connected to can have up to 1000 subscriptions. Each +> > subscription is for a key - this can be an SSK or a CHK - and if it is +> > an SSK, it can have the "persist" flag. If it does not have this flag, +> > or this flag is false, then when the data is found, the subscription is +> > removed. If we no do not have enough subscription quota left for a given +> > subscription, for that node, or for ourselves with another node, we +> > reject the request attempting to add the subscription with a +> > RejectOverload. Nodes can unsubscribe. Each subscription can have an +> > expiry date (actually, perhaps we should impose one). +> > +> > Passive requests can only be created by actual requests with the passive +> > request flag. The request will be routed as normal, until HTL runs out +> > (or it finds the data if the TUK flag is not enabled). +> > +> > Now, how do we make this scale?: +> > 1. If the TUK flag is off, and the request reaches a node which has the +> > data, we return it, and the request ends. No passive requests are +> > installed. +> > 2. If any request for a TUK, even if not passive, reaches a node which +> > already has a valid subscription at a higher or equal HTL for the key, +> > then the request ends at that point, and returns the data that node has. +> > If the passive-request flag is enabled, then passive request tags are +> > added up to that node, and that node adds one for the node connecting to +> > it if necessary. +> > 3. What to do about looping and coalescing? One option is to not +> > coalesce at all with running requests, and either forward the request +> > as-is or reject it... more in my next email... +> +> As far as looping and coalescing goes: We can coalesce the requests +> exactly as proposed for regular requests. So the request will not get +> stuck by eating its own tail. As long as the network isn't totally +> degenerate this shouldn't be a big problem. Also, if we do hit our own +> tail, we could have a "lightweight subscribe" which isn't refcounted, +> but passes packets both ways to strengthen the network. +> +> As far as resubscription and renewals go, there are two options really: +> 1. Try to get the network to maintain the subscriptions itself; +> automatic resubscription etc. +> 2. Make the client resubscribe itself every so often. We would have to +> let some of their requests through... +> 3. Don't bother. Switch streams from time to time at the client level. +> The problem with this is that we will still have to deal with situations +> such as the parent of a node being disconnected. + +Which we can do by simply telling our clients/dependants to resubscribe! +-- +Matthew J Toseland - [EMAIL PROTECTED] +Freenet Project Official Codemonkey - http://freenetproject.org/ +ICTHUS - Nothing is impossible. Our Boss says so. + + +To: [email protected] +Subject: Re: [Tech] Pub/sub = passive requests + TUKs +From: Matthew Toseland <[EMAIL PROTECTED]> +X-BeenThere: [email protected] +X-Mailman-Version: 2.1.5 +Reply-To: [email protected] +List-Id: tech.freenetproject.org +List-Unsubscribe: <http://emu.freenetproject.org/cgi-bin/mailman/listinfo/tech>, + <mailto:[EMAIL PROTECTED]> +List-Archive: <http://emu.freenetproject.org/pipermail/tech> +List-Post: <mailto:[email protected]> +List-Help: <mailto:[EMAIL PROTECTED]> +List-Subscribe: <http://emu.freenetproject.org/cgi-bin/mailman/listinfo/tech>, + <mailto:[EMAIL PROTECTED]> + +On Thu, Oct 20, 2005 at 07:56:56PM +0100, Matthew Toseland wrote: +> On Thu, Oct 20, 2005 at 07:46:46PM +0100, Matthew Toseland wrote: +> > On Thu, Oct 20, 2005 at 06:54:42PM +0100, Matthew Toseland wrote: +> > > Suppose: +> > > - We have a separate store for SSKs. They have 1kB of payload, plus the +> > > headers. +> > > - SSKs are versioned, with a timestamp. I.e. they become TUKs. Requests +> > > can optionally propagate far enough to fetch all values. There can be +> > > any number of SSKs with the same key and different version numbers / +> > > datestamps. Requests can ask for either a single specific version, the +> > > most recent version, or everything since a given revision. Only the +> > > data blocks which are actually returned are promoted in the cache. +> > > - To subscribe to a stream, I send a request out for the key I want, +> > > with the minimum acceptable version number/datestamp, plus three flags: +> > > - Passive request - If the data is not found, keep a tag on each node +> > > on the chain, and if it ever passes through any of the nodes on the +> > > chain, pass it back to me. +> > > - TUK - Return the most recent version of the SSK. +> > > - Subscribe - Requires the other two. Means the tag persists. +> > > Obviously it will be deleted if the originator rejects the data, or +> > > unsubscribes. +> > > +> > > Now, how would this actually work? +> > > +> > > Each node we are connected to can have up to 1000 subscriptions. Each +> > > subscription is for a key - this can be an SSK or a CHK - and if it is +> > > an SSK, it can have the "persist" flag. If it does not have this flag, +> > > or this flag is false, then when the data is found, the subscription is +> > > removed. If we no do not have enough subscription quota left for a given +> > > subscription, for that node, or for ourselves with another node, we +> > > reject the request attempting to add the subscription with a +> > > RejectOverload. Nodes can unsubscribe. Each subscription can have an +> > > expiry date (actually, perhaps we should impose one). +> > > +> > > Passive requests can only be created by actual requests with the passive +> > > request flag. The request will be routed as normal, until HTL runs out +> > > (or it finds the data if the TUK flag is not enabled). +> > > +> > > Now, how do we make this scale?: +> > > 1. If the TUK flag is off, and the request reaches a node which has the +> > > data, we return it, and the request ends. No passive requests are +> > > installed. +> > > 2. If any request for a TUK, even if not passive, reaches a node which +> > > already has a valid subscription at a higher or equal HTL for the key, +> > > then the request ends at that point, and returns the data that node has. +> > > If the passive-request flag is enabled, then passive request tags are +> > > added up to that node, and that node adds one for the node connecting to +> > > it if necessary. +> > > 3. What to do about looping and coalescing? One option is to not +> > > coalesce at all with running requests, and either forward the request +> > > as-is or reject it... more in my next email... +> > +> > As far as looping and coalescing goes: We can coalesce the requests +> > exactly as proposed for regular requests. So the request will not get +> > stuck by eating its own tail. As long as the network isn't totally +> > degenerate this shouldn't be a big problem. Also, if we do hit our own +> > tail, we could have a "lightweight subscribe" which isn't refcounted, +> > but passes packets both ways to strengthen the network. +> > +> > As far as resubscription and renewals go, there are two options really: +> > 1. Try to get the network to maintain the subscriptions itself; +> > automatic resubscription etc. +> > 2. Make the client resubscribe itself every so often. We would have to +> > let some of their requests through... +> > 3. Don't bother. Switch streams from time to time at the client level. +> > The problem with this is that we will still have to deal with situations +> > such as the parent of a node being disconnected. +> +> Which we can do by simply telling our clients/dependants to resubscribe! + +So the choice is this: either +a) When something breaks, everyone has to resubscribe, or +b) When something breaks, we tell all dependant nodes to let our +requests through, and resubscribe ourself. + +Either way there has to be a concept of "tree" - the node which we +expect to get messages from (the node we routed to). We don't have to +adhere strictly to this in passing data packets onwards, but it does +have to exist. Therefore, we will have a concept of "root", and of "root +location". So a certain amount of what has been planned has to be kept. +A lot of it, in fact, but the data storage is different, and we can have +very many streams. + +So: + +If we lose our parent: +- Simple way: We forcibly unsubscribe all our dependants. Their clients + will resubscribe. +- Hard way: We send a SubscribeRestarted to all our dependants. Then we + send a request out, similar to the one we have previously received, + seeking the key. It has a key-must-be-better-than property, as do + regular requests (they use it in combination with HTL); this is unset, + or rather it is set far away from the target. +-- +Matthew J Toseland - [EMAIL PROTECTED] +Freenet Project Official Codemonkey - http://freenetproject.org/ +ICTHUS - Nothing is impossible. Our Boss says so. + + +To: [email protected] +Subject: Re: [Tech] Pub/sub = passive requests + TUKs +From: Matthew Toseland <[EMAIL PROTECTED]> +X-BeenThere: [email protected] +X-Mailman-Version: 2.1.5 +Reply-To: [email protected] +List-Id: tech.freenetproject.org +List-Unsubscribe: <http://emu.freenetproject.org/cgi-bin/mailman/listinfo/tech>, + <mailto:[EMAIL PROTECTED]> +List-Archive: <http://emu.freenetproject.org/pipermail/tech> +List-Post: <mailto:[email protected]> +List-Help: <mailto:[EMAIL PROTECTED]> +List-Subscribe: <http://emu.freenetproject.org/cgi-bin/mailman/listinfo/tech>, + <mailto:[EMAIL PROTECTED]> + +[20:16] <toad_> linyos: can you very briefly summarize the reason for +not just limiting it to a low number of streams and caching them in RAM? +[20:17] <linyos> yes, that should not be done because it limits you to a +low number of streams. +[20:18] <toad_> but also, we will want to implement TUKs and passive +requests at some point +[20:18] <linyos> users will probably be very enthusiastic about keeping +tons of streams open. + +On Thu, Oct 20, 2005 at 08:17:04PM +0100, Matthew Toseland wrote: +> On Thu, Oct 20, 2005 at 07:56:56PM +0100, Matthew Toseland wrote: +> > On Thu, Oct 20, 2005 at 07:46:46PM +0100, Matthew Toseland wrote: +> > > On Thu, Oct 20, 2005 at 06:54:42PM +0100, Matthew Toseland wrote: +> > > > Suppose: +> > > > - We have a separate store for SSKs. They have 1kB of payload, plus the +> > > > headers. +> > > > - SSKs are versioned, with a timestamp. I.e. they become TUKs. Requests +> > > > can optionally propagate far enough to fetch all values. There can be +> > > > any number of SSKs with the same key and different version numbers / +> > > > datestamps. Requests can ask for either a single specific version, the +> > > > most recent version, or everything since a given revision. Only the +> > > > data blocks which are actually returned are promoted in the cache. +> > > > - To subscribe to a stream, I send a request out for the key I want, +> > > > with the minimum acceptable version number/datestamp, plus three ++flags: +> > > > - Passive request - If the data is not found, keep a tag on each node +> > > > on the chain, and if it ever passes through any of the nodes on the +> > > > chain, pass it back to me. +> > > > - TUK - Return the most recent version of the SSK. +> > > > - Subscribe - Requires the other two. Means the tag persists. +> > > > Obviously it will be deleted if the originator rejects the data, or +> > > > unsubscribes. +> > > > +> > > > Now, how would this actually work? +> > > > +> > > > Each node we are connected to can have up to 1000 subscriptions. Each +> > > > subscription is for a key - this can be an SSK or a CHK - and if it is +> > > > an SSK, it can have the "persist" flag. If it does not have this flag, +> > > > or this flag is false, then when the data is found, the subscription is +> > > > removed. If we no do not have enough subscription quota left for a given +> > > > subscription, for that node, or for ourselves with another node, we +> > > > reject the request attempting to add the subscription with a +> > > > RejectOverload. Nodes can unsubscribe. Each subscription can have an +> > > > expiry date (actually, perhaps we should impose one). +> > > > +> > > > Passive requests can only be created by actual requests with the passive +> > > > request flag. The request will be routed as normal, until HTL runs out +> > > > (or it finds the data if the TUK flag is not enabled). +> > > > +> > > > Now, how do we make this scale?: +> > > > 1. If the TUK flag is off, and the request reaches a node which has the +> > > > data, we return it, and the request ends. No passive requests are +> > > > installed. +> > > > 2. If any request for a TUK, even if not passive, reaches a node which +> > > > already has a valid subscription at a higher or equal HTL for the key, +> > > > then the request ends at that point, and returns the data that node has. +> > > > If the passive-request flag is enabled, then passive request tags are +> > > > added up to that node, and that node adds one for the node connecting to +> > > > it if necessary. +> > > > 3. What to do about looping and coalescing? One option is to not +> > > > coalesce at all with running requests, and either forward the request +> > > > as-is or reject it... more in my next email... +> > > +> > > As far as looping and coalescing goes: We can coalesce the requests +> > > exactly as proposed for regular requests. So the request will not get +> > > stuck by eating its own tail. As long as the network isn't totally +> > > degenerate this shouldn't be a big problem. Also, if we do hit our own +> > > tail, we could have a "lightweight subscribe" which isn't refcounted, +> > > but passes packets both ways to strengthen the network. +> > > +> > > As far as resubscription and renewals go, there are two options really: +> > > 1. Try to get the network to maintain the subscriptions itself; +> > > automatic resubscription etc. +> > > 2. Make the client resubscribe itself every so often. We would have to +> > > let some of their requests through... +> > > 3. Don't bother. Switch streams from time to time at the client level. +> > > The problem with this is that we will still have to deal with situations +> > > such as the parent of a node being disconnected. +> > +> > Which we can do by simply telling our clients/dependants to resubscribe! +> +> So the choice is this: either +> a) When something breaks, everyone has to resubscribe, or +> b) When something breaks, we tell all dependant nodes to let our +> requests through, and resubscribe ourself. +> +> Either way there has to be a concept of "tree" - the node which we +> expect to get messages from (the node we routed to). We don't have to +> adhere strictly to this in passing data packets onwards, but it does +> have to exist. Therefore, we will have a concept of "root", and of "root +> location". So a certain amount of what has been planned has to be kept. +> A lot of it, in fact, but the data storage is different, and we can have +> very many streams. +> +> So: +> +> If we lose our parent: +> - Simple way: We forcibly unsubscribe all our dependants. Their clients +> will resubscribe. +> - Hard way: We send a SubscribeRestarted to all our dependants. Then we +> send a request out, similar to the one we have previously received, +> seeking the key. It has a key-must-be-better-than property, as do +> regular requests (they use it in combination with HTL); this is unset, +> or rather it is set far away from the target. +> -- +> Matthew J Toseland - [EMAIL PROTECTED] +> Freenet Project Official Codemonkey - http://freenetproject.org/ +> ICTHUS - Nothing is impossible. Our Boss says so. + + + +> _______________________________________________ +> Tech mailing list +> [email protected] +> http://emu.freenetproject.org/cgi-bin/mailman/listinfo/tech + +-- +Matthew J Toseland - [EMAIL PROTECTED] +Freenet Project Official Codemonkey - http://freenetproject.org/ +ICTHUS - Nothing is impossible. Our Boss says so. Modified: trunk/freenet/devnotes/pubsub/linyos-on-pubsub.txt =================================================================== --- trunk/freenet/devnotes/pubsub/linyos-on-pubsub.txt 2005-10-21 13:07:28 UTC (rev 7440) +++ trunk/freenet/devnotes/pubsub/linyos-on-pubsub.txt 2005-10-21 13:17:13 UTC (rev 7441) @@ -1,97 +1,7 @@ -_> :) -[15:26] <linyos> great minds think alike. -[15:26] <linyos> i thought of that too. -[15:27] <linyos> though not so much for wiring money across the darknet graph as for paying for services (like message-forwarding) -[15:27] <toad_> ah -[15:29] <toad_> well yes there is that too but personally i think that introducing markets in bandwidth is suicide as bandwidth doesn't really fit into the traditional scarcity model, and more importantly creating a market like that will generally grossly distort routing and destroy and possible scalability properties... better to provide *basic* forwarding services as a quid-pro-quo part of being on the network... it's vital to your anonymity anyway -[15:29] <toad_> now, if you want higher level services, like buying webspace on the real internet, that DOES require payment -[15:30] <toad_> also, a lot of the security problems go away if the currency is an internal one which is not inherently exchangeable, e.g. LETS -[15:30] <toad_> you shift them to the inevitable exchange traders, of course -[15:30] <linyos> right -[15:30] <toad_> but that's not your problem as long as you have a lot of stuff going on internally -[15:31] <toad_> actually it's quite interesting from a LETS angle as LETS has never been able to scale, nor is it intended to... mostly because it's a trust system. but the interesting thing here is that we have a distributed, scalable trust system... -[15:32] * toad_ thinks it's worth playing with, one day, after we've shipped freenet 1.0 ;) -[15:32] <toad_> linyos: just curious, which statement is the "right" to? -[15:33] <linyos> that people would buy and sell them for other assets if at all possible. -[15:34] <toad_> yeah, there will always be exchange services on any nontrivial internal currency -[15:34] <linyos> unless you design the system to preclude it. -[15:34] <linyos> which is surely possible. -[15:34] <toad_> but imho it makes sense for an internal currency _not_ to be by nature an external one, for various reasons -[15:35] <toad_> linyos: it is? -[15:36] <linyos> you caught me, i can't prove that. -[15:36] <linyos> just a guess. -[15:36] <toad_> i doubt it very much -[15:37] <toad_> anything which isn't completely a toy currency (e.g. the ena on frost) will be exchangeable, it's just a matter of how easy it is to do that -[15:37] <linyos> well, suppose the host that earned the money has to spend it itself. -[15:38] <linyos> i guess you could sell extra-network connections to your in-network host. -[15:38] <-- sleon|work has left this server. (Remote closed the connection) -[15:39] <linyos> ie, ssh to my node and for ten bucks you can make requests with all the funny-money i've earned. -[15:39] <linyos> bbiab. -[15:39] <toad_> :) [15:39] <toad_> hmmm [15:40] <toad_> should I implement CoalesceNotify for requests/inserts first, or should I continue with pub/sub? [15:43] <toad_> hmmm i see... -[16:17] <-- Ash-Fox has left this server. (Remote closed the connection) -[16:18] --> Danar has joined this channel. ([EMAIL PROTECTED]/session/x-781f4d1bf5d423ca) -[16:18] <-- FallingBuzzard has left this channel. () -[16:19] <Danar> the faq & wiki say to open freenet.conf, but don't say where it is, and i can't seem to find it -[16:19] --> Ash-Fox has joined this channel. ([EMAIL PROTECTED]) -[16:20] <Danar> what's wrong with the debian package? -[16:32] <nextgens> hi -[16:32] * nextgens will have an inet. connection at home soon :) -[16:37] <sandos> good =) -[16:37] <sandos> I couldnt do without one.. : -[16:38] <nextgens> I managed to ... but I agree; that's hard ;) -[17:01] --> NullAcht15 has joined this channel. ([EMAIL PROTECTED]) -[17:05] <Danar> what's wrong with the debian package? -[17:07] <linyos> i don't know, but there is no reason not to use the latest snapshot, anyway. -[17:10] <nextgens> Danar: they are 3 years old ;) -[17:13] --> Sugadude has joined this channel. ([EMAIL PROTECTED]/session/x-561652a36c2ac246) -[17:13] --> FallingBuzzard has joined this channel. ([EMAIL PROTECTED]) -[17:14] --> Romster2 has joined this channel. ([EMAIL PROTECTED]/session/x-937bf78089dc33e3) -[17:15] <Danar> i see -[17:15] <Danar> the faq & wiki say to open freenet.conf, but don't say where it is, and i can't seem to find it -[17:16] <Sugadude> Danar: On windows it's where you installed Freenet and it's called freenet.ini -[17:16] <Danar> why doesn't someone contact the package maintainer? -[17:16] <toad_> the current freenet only runs on the proprietary sun java :| -[17:16] <Danar> Sugadude, i know. :p -[17:16] <-- FallingBuzzard has left this channel. () -[17:16] <toad_> of course that shouldn't stop it from going into Contrib... -[17:17] * Sugadude missed most of the conversation, sorry. :P -[17:17] <Danar> Sugadude, i need to know where freenet.conf is -[17:17] <Danar> ;) -[17:18] <Sugadude> I would guess it's where you installed freenet. -[17:18] <toad_> Danar: are you using the debian package? -[17:18] <Danar> yeah... :/ -[17:18] <toad_> if so, it's in /etc/freenet/freenet* -[17:18] <toad_> however, i recommend you purge the debian package -[17:18] <toad_> and install from scratch -[17:19] <Sugadude> Is Freenet in Debian stable? Is it version 50x? ;) -[17:19] <greycat> NEVER EVER EVER use the Freenet .deb packages. -[17:19] <greycat> Ever. -[17:19] <Danar> i had installed the testing version -[17:19] <Sugadude> Yeah, so it's probably Freenet 51x. :) -[17:20] <Danar> then what's the point of having the package? why doesn't someone contact the maintainer? or become the maintainer or something? -[17:21] <Sugadude> Danar: You're more than welcome to. -[17:21] <linyos> hm, no freenet package in ubuntu -[17:22] <Danar> well thanks -[17:22] <-- Danar has left this server. ("Client exiting") -[17:22] <Sugadude> linyos: Just add the main debian repository manually. -[17:22] <toad_> really, there ought to be a real debian package... -[17:22] --> Hory has joined this channel. ([EMAIL PROTECTED]) -[17:22] <toad_> even if it's in contrib and depends on the proprietary sun packages -[17:22] <toad_> other packages do this -[17:22] <toad_> hi Hory -[17:23] <Hory> hi -[17:26] <linyos> i wonder if you could beat the linux kernel into routing IP packets over ipsec-encrypted darknet tunnels. -[17:27] <toad_> probably -[17:27] <toad_> that's just VPN really -[17:28] <toad_> there is an interface for userspace networking devices... -[17:28] <linyos> i mean implement greedy small-world routing in there -[17:29] <toad_> ahhh -[17:29] <linyos> or something -[17:29] <toad_> that would be cool -[17:29] <toad_> it would probably still have to be partly userspace +... [17:29] <toad_> okay, where was i? [17:30] <toad_> if A sends a subscribe request to B [17:30] <toad_> and B forwards it _______________________________________________ cvs mailing list [email protected] http://emu.freenetproject.org/cgi-bin/mailman/listinfo/cvs
