Hmmm ... gossiptorrent?
On Wed, Oct 21, 2009 at 7:23 AM, Marcus Herou <[email protected]> wrote: > > Hi. > > We host a search app which is based on feeds of blogs/twitter/forums/ > news etc. We are as you are mentioning polling everything like crazy > and it seems like a total waste of everyones resources. > > So this means that subscribing to something which would potentially > deliver the material to us would be great not just for us but as well > all sites we are crawling. > > However who would like to open up a firehose for free for everyone to > consume ? It will for sure consume a lot of bandwidth and a few > subscribers will consume most of the bandwidth with this model. > I thought of something that might solve this issue. Consider the > following: > > 1) > * Charge for the bandwidth (wordpress.com does this with flat fee) > > 2) > * Everyone that have firehose consuming needs should as well start a > hub to show good faith and morale. > * Add support in firehose enabled hubs to share state (with a > master ?) > * A firehose enabled hub can subscribe to a master hub which makes > sure that the subscriber as well fulfils some form of contract (i.e. > actually updating/delivering feeds) > * Each firehose enabled hub must be public and everyone can subscribe > to the feeds like as of current. > * To share load equally (morale part) then subscribers should > subscribe to a loadbalanced dns name or some form of delegate > lb.pshb.com = master hub > Example 1: lb.pshb.com resolves to pshb.tailsweep.com > pshb.google.com, effectively DNS-roundrobin > Example 2: lb.pshb.com delegates to any active master connected hub > in some way. > > This might be too complex to implement and bottlenecks occur at the > master but systems like Hadoop have bottlenecks in terms of the > NameNode (master) and it seems to perform just perfect so it can be > done. However each firehose hub probably need to persist each feed for > a certain amount of time before purging it. > > Anyway this was just a thought. We at Tailsweep probably could help in > making this happen if there exists some interest. > > Cheers > > //Marcus > > > > > > On Oct 20, 8:41 pm, Bob Wyman <[email protected]> wrote: >> On Tue, Oct 20, 2009 at 11:22 AM, igrigorik <[email protected]> wrote: >> > Specifically, if we treat 'firehose' as any bundle of >> > feeds (all, or some), then a hub could define >> > multiple firehose streams. >> >> There should be no question that there is tremendous utility in being able >> to compose all sorts of "bundles" of topics into distinct feeds. It is >> probably also the case that we can identify some number of such bundles that >> would be useful to a large number of subscribers. On the other hand, many >> bundles will be very specific and only useful to one or a small number of >> subscribers. In fact, I think what we'll see is that once we have the core >> PSHB defined, we'll then see innovation in the definition of "down stream" >> services whose function is precisely to build and deliver such bundles. Some >> of these services will aggregate groups of topics while others will focus >> instead on creating content-based streams -- they will bundle together >> individual entries based on the content of those entries rather than simply >> combining all entries from some set of topics. >> >> I think we should be careful not to force too much of the burden of bundling >> or aggregating into the core PSHB hub specification. If we want to address >> the challenges of building bundles or aggregations, I think it best to do so >> in secondary or companion specifications. This will keep the core cleaner >> and easy to understand while also allowing the core to be deployed without >> being delayed by discussions over non-core issues. >> >> Having argued against making the core more complicated by extending it to >> include creating aggregate topics, I still suggest that it would be useful >> to have the core system define a common means to obtain a pure "firehose" >> feed of all topics. The current hub spec works for people who only want >> "none or some" of the topics served by the hub. I suggest that we expand >> this to have hubs know how to provide "none, some or all" of the topics. >> The reason for adding support of "all topics" is that we know, without much >> question, that such an "all topics" feed will be required by many of the >> downstream services that we will one day be relying on to create more finely >> defined aggregations. Given that this specific feed will be commonly >> required, it would be best if we had a common mechanism for a downstream >> service/subscriber to request that feed and that we set some expectations >> for how that feed will be formatted and delivered (i.e. Atom entries, >> persistent connections, chunked content model, ...). It would be very >> cumbersome for a downstream filtering/aggregating service to need to puzzle >> through service specific mechanisms for discovering how to obtain a firehose >> feed of "all topics" from many different hubs. >> >> bob wyman >> >> On Tue, Oct 20, 2009 at 11:22 AM, igrigorik <[email protected]> wrote: >> >> > Right, so how does the smart hub aggregate the feeds? Does it then >> > have to crawl to find the list? That wouldn't be very useful. Having >> > said that... >> >> > +1 For 'smart, aggregating hub generating a synthetic feed' >> > +1 For XRD discovery of the firehose endpoint. >> >> > Thinking a bit more about the firehose, what about making it more >> > flexible. Specifically, if we treat 'firehose' as any bundle of feeds >> > (all, or some), then a hub could define multiple firehose streams. For >> > example, at PostRank we classify feeds by topic, so if someone wanted >> > to subscribe to "Technology", we could expose that as a firehose so >> > the user doesn't have to subscribe to every feed in that topic. In >> > essence, a firehose stream is then any bundle of feeds. >> >> > This may be overloading the hub spec but the overall mechanics would >> > be: >> > - A (super)user can declare a firehose endpoint >> > - A (super)user is then able to add or remove subscriptions from the >> > firehose to create arbitrary aggregation streams >> > - A subscriber uses XRD to discover the available aggregation streams >> > - Firehose with 'all' feeds is a special case of the above, where all >> > feeds are present >> >> > This definitely adds more complexity into the hub... The alternative >> > is of course for the publisher to create a syndicated feed and publish >> > that directly as a standalone feed. Still trying to weight the up/ >> > downsides in my head, but want to put it out there as an idea. >> >> > -------- >> > Ilya Grigorik >> > postrank.com >
