I'm evaluating pubsubhubbub for use as a message queue. The messages have multiple receivers (i.e. subscribers) and all receivers should receive all messages, so on the face off it, it seems like a good match.
I plan to use feed-windowing with ETag/If-None-Match to publish chunks of new/updated entries, so I'm essentially doing a manual "fat ping" for publishing: I always know exactly what I'm going to deliver when I ping the server, and any one update will only be included once in the feed (are there any plans to add real fat pings from the publisher to the hub?). The messages describe additions and updates to a set of appr. 5 million entries. The entries describe access rights, so I need to be pretty sure that all messages reach their subscribers. Todd Hoff mentions in an earlier thread[1] that pubsubhubbub "seems most useful in the lossy append style event scenario". Unlike the example in that thread, my situation fits nicely with a "append style event scenario", but not with the lossy part. I would love to hear from other people who have built non-lossy systems with pubsubhubbub. My primary concern is that some updates/inserts will be lost: As each update/insert is only included once, I need to be fairly sure that the hub will never discard anything. Is this likely to happen, for example in the app engine implementation? What happens if a subscriber isn't able to receive messages fast enough during a peak period? At what point would the hub start discarding messages? Secondly, how good is throughput in the app engine implementation? We need to handle at least 10 MB per minute at peak, I imagine that this wouldn't be a problem? [1]: http://groups.google.com/group/pubsubhubbub/browse_thread/thread/55ba7a29aa800ab4/7ff09f89192aed3f?lnk=gst
