Feedtree looks cool.... but updated 2006 ? On Wed, Oct 21, 2009 at 2:20 PM, Nick Johnson (Google) < [email protected]> wrote:
> On Wed, Oct 21, 2009 at 1:14 PM, Alexis Richardson < > [email protected]> wrote: > >> >> Hmmm ... gossiptorrent? >> > > Feedtree. > > >> >> On Wed, Oct 21, 2009 at 7:23 AM, Marcus Herou >> <[email protected]> wrote: >> > >> > Hi. >> > >> > We host a search app which is based on feeds of blogs/twitter/forums/ >> > news etc. We are as you are mentioning polling everything like crazy >> > and it seems like a total waste of everyones resources. >> > >> > So this means that subscribing to something which would potentially >> > deliver the material to us would be great not just for us but as well >> > all sites we are crawling. >> > >> > However who would like to open up a firehose for free for everyone to >> > consume ? It will for sure consume a lot of bandwidth and a few >> > subscribers will consume most of the bandwidth with this model. >> > I thought of something that might solve this issue. Consider the >> > following: >> > >> > 1) >> > * Charge for the bandwidth (wordpress.com does this with flat fee) >> > >> > 2) >> > * Everyone that have firehose consuming needs should as well start a >> > hub to show good faith and morale. >> > * Add support in firehose enabled hubs to share state (with a >> > master ?) >> > * A firehose enabled hub can subscribe to a master hub which makes >> > sure that the subscriber as well fulfils some form of contract (i.e. >> > actually updating/delivering feeds) >> > * Each firehose enabled hub must be public and everyone can subscribe >> > to the feeds like as of current. >> > * To share load equally (morale part) then subscribers should >> > subscribe to a loadbalanced dns name or some form of delegate >> > lb.pshb.com = master hub >> > Example 1: lb.pshb.com resolves to pshb.tailsweep.com >> > pshb.google.com, effectively DNS-roundrobin >> > Example 2: lb.pshb.com delegates to any active master connected hub >> > in some way. >> > >> > This might be too complex to implement and bottlenecks occur at the >> > master but systems like Hadoop have bottlenecks in terms of the >> > NameNode (master) and it seems to perform just perfect so it can be >> > done. However each firehose hub probably need to persist each feed for >> > a certain amount of time before purging it. >> > >> > Anyway this was just a thought. We at Tailsweep probably could help in >> > making this happen if there exists some interest. >> > >> > Cheers >> > >> > //Marcus >> > >> > >> > >> > >> > >> > On Oct 20, 8:41 pm, Bob Wyman <[email protected]> wrote: >> >> On Tue, Oct 20, 2009 at 11:22 AM, igrigorik <[email protected]> wrote: >> >> > Specifically, if we treat 'firehose' as any bundle of >> >> > feeds (all, or some), then a hub could define >> >> > multiple firehose streams. >> >> >> >> There should be no question that there is tremendous utility in being >> able >> >> to compose all sorts of "bundles" of topics into distinct feeds. It is >> >> probably also the case that we can identify some number of such bundles >> that >> >> would be useful to a large number of subscribers. On the other hand, >> many >> >> bundles will be very specific and only useful to one or a small number >> of >> >> subscribers. In fact, I think what we'll see is that once we have the >> core >> >> PSHB defined, we'll then see innovation in the definition of "down >> stream" >> >> services whose function is precisely to build and deliver such bundles. >> Some >> >> of these services will aggregate groups of topics while others will >> focus >> >> instead on creating content-based streams -- they will bundle together >> >> individual entries based on the content of those entries rather than >> simply >> >> combining all entries from some set of topics. >> >> >> >> I think we should be careful not to force too much of the burden of >> bundling >> >> or aggregating into the core PSHB hub specification. If we want to >> address >> >> the challenges of building bundles or aggregations, I think it best to >> do so >> >> in secondary or companion specifications. This will keep the core >> cleaner >> >> and easy to understand while also allowing the core to be deployed >> without >> >> being delayed by discussions over non-core issues. >> >> >> >> Having argued against making the core more complicated by extending it >> to >> >> include creating aggregate topics, I still suggest that it would be >> useful >> >> to have the core system define a common means to obtain a pure >> "firehose" >> >> feed of all topics. The current hub spec works for people who only want >> >> "none or some" of the topics served by the hub. I suggest that we >> expand >> >> this to have hubs know how to provide "none, some or all" of the >> topics. >> >> The reason for adding support of "all topics" is that we know, without >> much >> >> question, that such an "all topics" feed will be required by many of >> the >> >> downstream services that we will one day be relying on to create more >> finely >> >> defined aggregations. Given that this specific feed will be commonly >> >> required, it would be best if we had a common mechanism for a >> downstream >> >> service/subscriber to request that feed and that we set some >> expectations >> >> for how that feed will be formatted and delivered (i.e. Atom entries, >> >> persistent connections, chunked content model, ...). It would be very >> >> cumbersome for a downstream filtering/aggregating service to need to >> puzzle >> >> through service specific mechanisms for discovering how to obtain a >> firehose >> >> feed of "all topics" from many different hubs. >> >> >> >> bob wyman >> >> >> >> On Tue, Oct 20, 2009 at 11:22 AM, igrigorik <[email protected]> wrote: >> >> >> >> > Right, so how does the smart hub aggregate the feeds? Does it then >> >> > have to crawl to find the list? That wouldn't be very useful. Having >> >> > said that... >> >> >> >> > +1 For 'smart, aggregating hub generating a synthetic feed' >> >> > +1 For XRD discovery of the firehose endpoint. >> >> >> >> > Thinking a bit more about the firehose, what about making it more >> >> > flexible. Specifically, if we treat 'firehose' as any bundle of feeds >> >> > (all, or some), then a hub could define multiple firehose streams. >> For >> >> > example, at PostRank we classify feeds by topic, so if someone wanted >> >> > to subscribe to "Technology", we could expose that as a firehose so >> >> > the user doesn't have to subscribe to every feed in that topic. In >> >> > essence, a firehose stream is then any bundle of feeds. >> >> >> >> > This may be overloading the hub spec but the overall mechanics would >> >> > be: >> >> > - A (super)user can declare a firehose endpoint >> >> > - A (super)user is then able to add or remove subscriptions from the >> >> > firehose to create arbitrary aggregation streams >> >> > - A subscriber uses XRD to discover the available aggregation >> streams >> >> > - Firehose with 'all' feeds is a special case of the above, where >> all >> >> > feeds are present >> >> >> >> > This definitely adds more complexity into the hub... The alternative >> >> > is of course for the publisher to create a syndicated feed and >> publish >> >> > that directly as a standalone feed. Still trying to weight the up/ >> >> > downsides in my head, but want to put it out there as an idea. >> >> >> >> > -------- >> >> > Ilya Grigorik >> >> > postrank.com >> > >> > > > > -- > Nick Johnson, Developer Programs Engineer, App Engine > Google Ireland Ltd. :: Registered in Dublin, Ireland, Registration Number: > 368047 > -- Marcus Herou CTO and co-founder Tailsweep AB +46702561312 [email protected] http://www.tailsweep.com/
