right, i can't really tell either. it looks like the churn of incoming requests is powering the "background" workers, the real change is the async proxy (AsyncAPIProxy.start_call) to fire the workers off without affecting the incoming request (at least I think thats what is happening...)
it would be nice to see the code behind this import (since it contains the AsyncRPC class) from google3.apphosting.runtime import _apphosting_runtime___python__apiproxy cheers brian On Mar 11, 6:17 pm, peterk <[email protected]> wrote: > The app is actually live here: > > http://pubsubhubbub.appspot.com/http://pubsubhubbub-subscriber.appspot.com/ > > (pubsubhubbub-publisher isn't there, but it's trivial to upload your > own.) > > This suggests it's working on appengine as it is now. Been looking > through the source, and I'm not entirely clear on how the 'background > workers' are actually working..there are two, one for pulling updates > to feeds from publishers, and one for propogating updates to > subscribers in batches. > > But like I say, I can't see how they're actually started and running > constantly. There is a video here of a live demonstration: > > http://www.veodia.com/player.php?vid=fCNU1qQ1oSs > > The background workers seem to be behaving as desired there, but I'm > not sure if they were just constantly polling some urls to keep the > workers live for the purposes of that demo, or if they're actually > running somehow constantly on their own.. I can't actually get the > live app at the urls above to work, but not sure if it's because > background workers aren't really working, or because i'm feeding it > incorrect urls/configuration etc. > > On Mar 11, 8:01 pm, bFlood <[email protected]> wrote: > > > great find peterk! cant help but notice the very interesting > > async_apiproxy.py code in that project either...async db and url calls > > would be awesome > > > cheers > > brian > > > On Mar 11, 3:22 pm, peterk <[email protected]> wrote: > > > > I've done a good bit of googling about what exactly it is I'm trying > > > to achieve, the correct terminology etc. and turns out I'm really > > > looking for an efficient way to do pub/sub, publication/subscription, > > > on appengine. > > > > Which led me to this eyebrow-raising little app from Brett Slakin and > > > Brad Fitzpatrick: > > > >http://code.google.com/p/pubsubhubbub/ > > > > Jaiku folk seem to be involved too. > > > > Basically they have an experimental pub/sub system running on GAE, > > > using http instead of xmpp. The goals are somewhat broader than my > > > requirements - basically you have an atom feed at one url you post > > > data to, the atom feed updates a hub, and the hub then updates your > > > subscribers at different urls, so it's all a lot more distributed than > > > I need. > > > > I imagine one could do something similar rolled into one application > > > that handles publishing, the hub, and subscription (e.g. like jaiku or > > > twitter). But the big minus point that sticks out here for me in > > > pubsubhubub is that it completely avoids the question of the total > > > write-cost per update. It can afford to do that because subscribers > > > here are distributed all over the place on different domains and > > > servers etc. so the write load gets spread across all those different > > > urls/machines..it's not the concern of the hub running on appengine. > > > But if pubs/subs/hub are all rolled into one application on appengine, > > > then your application will be soaking up the total write cost. The hub > > > handles updating each subscriber in the background, I think, so it > > > does solve the issue of writing to n subscribers in the space of one > > > request..the publisher issues its update to the hub, returns > > > immediately, and then the hub partitions out the subscriber updates > > > and works on updating everyone over time. But it doesn't solve the > > > issue of monetary cost of doing n writes for n subscribers...if one > > > publisher had lots of subscribers (e.g. hundreds of thousands or > > > millions), each of their updates could cost $$$! > > > > But I guess one might be safe in assuming the average number of subs > > > per publisher will be much lower than that..and make the idea > > > financially feasible.. > > > > Anyway, I thought this project looked very interesting...it's giving > > > me a lot to chew on now.. > > > > On Mar 11, 3:50 pm, peterk <[email protected]> wrote: > > > > > Hmm. Good thinking Andrew. I've done a bit more digging along those > > > > lines..I don't think Jaiku has publically said if and what messaging > > > > protocols they might use, but there seems to be some speculation that > > > > it uses XMPP. > > > > > That might also neatly align with their comment that they are in some > > > > cases "extending" GAE in their port..and the recent news that XMPP is > > > > scheduled to come to GAE at some point in the next 6 months. Perhaps > > > > it's the work on the Jaiku port that's bringing that functionality... > > > > > I didn't know anything about messaging protocols or XMPP until I dug > > > > around after your post, so it's all new to me. I found a project > > > > purporting to provide XMPP functionality in advance of Google's > > > > official support, athttp://imified-demo.appspot.com/, but looking at > > > > the code for it, I cannot for the life of me see how this could be > > > > used to address the problems discussed in my original email. That > > > > project seems only to take instant messages via a POST to a URL and > > > > save them to the datastore..but there doesn't seem to be any of the > > > > 'instant distribution' of updates to people subscribed to your > > > > presence etc. that other XMPP software boast about. Using that xmpp > > > > project linked, there doesn't seem to be anything there about pushing > > > > updates to subscribed users etc. > > > > > I dug around the app gallery for other open source apps along similar > > > > lines..there are some would-be twitter/jaiku clones..all I've seen use > > > > non-messaging approaches, but they all make one compromise or another > > > > similar to solutions I'd come up with myself (e.g. taking the last 5 > > > > updates from friends and sorting them yourself in memory by > > > > date..which has its own issues, for example, if one user had made >5 > > > > updates recently..+ you'd inevitably hit a roadblock on all these > > > > reads beyond a certain number of friends..there's only so many friends > > > > you could do this for before it gets too slow). > > > > > Any thoughts/ideas...? > > > > > On Mar 11, 1:54 pm, Andrew Badera <[email protected]> wrote: > > > > > > you'd have to wonder if there's a push out to Gnip somewhere ... or if > > > > > protocol buffers are involved .. > > > > > > Thanks- > > > > > - Andy Badera > > > > > - [email protected] > > > > > - (518) 641-1280 > > > > > - Tech Valley Code Camp 2009.1:http://www.techvalleycodecamp.com/ > > > > > - Google me:http://www.google.com/search?q=andrew+badera > > > > > > On Wed, Mar 11, 2009 at 9:21 AM, peterk <[email protected]> wrote: > > > > > > > I just read on your blog (from January) the intention to release the > > > > > > appengine port of Jaiku as open source when the port is > > > > > > finished..but > > > > > > I was wondering if I could be so cheeky as to jump ahead with a > > > > > > couple > > > > > > of questions about it. The requirements of Jaiku seem to line up > > > > > > roughly similarly with issues I'm having in a slightly different > > > > > > context, that I'm finding pretty challenging to implement > > > > > > efficiently > > > > > > on app engine. > > > > > > > With your service, you seem to track updates for friends and other > > > > > > people I follow..so I might have a long list of people I'm > > > > > > following, > > > > > > and you feed me their updates. > > > > > > > How do you implement this on GAE? > > > > > > > I've been toying with a very similar problem for some time now. It > > > > > > seems to me you cannot chain together queries such as > > > > > > me.friends.updates.order(..) to get your friends' latest updates, > > > > > > for > > > > > > example. You can't make n writes to n update queues for n people > > > > > > following you, since writes are so costly. If I store my friends in > > > > > > a > > > > > > list of keys, this limits the number of friends I can query at a > > > > > > given > > > > > > time to 30. e.g. updates.all().filter('user IN', me.friends) is > > > > > > limited to 30 subqueries. I may have many more friends, so this > > > > > > approach doesn't seem to be sufficient. > > > > > > > I've been scratching my head over a similar problem for some time > > > > > > now, > > > > > > coming up with various hairbrained schemes that have been overly- > > > > > > complex, none of which deliver scalability to the nth degree. So I'd > > > > > > really, really, really appreciate any insight you could provide in > > > > > > implementing this kind of data model on GAE!! Many thanks! --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~----------~----~----~----~------~----~------~--~---
