Hey Filip, About the publisher -> hub fat pings. I believe the spec is 'minimal' because it is not assumed that the hub and the publisher have a tight relationship which would enable hubs to trust publishers. At superfeedr, we obviously implement this, but also let the publisher fat ping us with the use of a shared secret key. (similar to the signature mechanism used for hub -> subscriber)
Similarly, I believe the current spec (and the AppEngine implementation) are minimal too, in the sense that they will do some kind of retries, but may lose data if your subscribing endpoint is away for a too long period of time. At Superfeedr we also do retries, and we also allow for digest notifications : once a day the hub delivers the full content of the feed, so it can be used to sync. This way, even if the endpoint was down for a significant time, it can be synced daily. Finally, we also allow for retrieval of the content of a feed on the hub, which means that if your subscriber can't wait for the digest notification, it can ask for it. My point is that the GAE implementation is probably mature for the general usecase. If you have specific requirement, you may have to dive in the code :) On Nov 2, 12:32 pm, Filip Svendsen <[email protected]> wrote: > I'm evaluating pubsubhubbub for use as a message queue. The messages > have multiple receivers (i.e. subscribers) and all receivers should > receive all messages, so on the face off it, it seems like a good > match. > > I plan to use feed-windowing with ETag/If-None-Match to publish chunks > of new/updated entries, so I'm essentially doing a manual "fat ping" > for publishing: I always know exactly what I'm going to deliver when I > ping the server, and any one update will only be included once in the > feed (are there any plans to add real fat pings from the publisher to > the hub?). > > The messages describe additions and updates to a set of appr. 5 > million entries. The entries describe access rights, so I need to be > pretty sure that all messages reach their subscribers. Todd Hoff > mentions in an earlier thread[1] that pubsubhubbub "seems most useful > in the lossy append style event scenario". Unlike the example in that > thread, my situation fits nicely with a "append style event scenario", > but not with the lossy part. I would love to hear from other people > who have built non-lossy systems with pubsubhubbub. > > My primary concern is that some updates/inserts will be lost: As each > update/insert is only included once, I need to be fairly sure that the > hub will never discard anything. Is this likely to happen, for example > in the app engine implementation? What happens if a subscriber isn't > able to receive messages fast enough during a peak period? At what > point would the hub start discarding messages? > > Secondly, how good is throughput in the app engine implementation? We > need to handle at least 10 MB per minute at peak, I imagine that this > wouldn't be a problem? > > [1]:http://groups.google.com/group/pubsubhubbub/browse_thread/thread/55ba...
