great find peterk! cant help but notice the very interesting async_apiproxy.py code in that project either...async db and url calls would be awesome
cheers brian On Mar 11, 3:22 pm, peterk <[email protected]> wrote: > I've done a good bit of googling about what exactly it is I'm trying > to achieve, the correct terminology etc. and turns out I'm really > looking for an efficient way to do pub/sub, publication/subscription, > on appengine. > > Which led me to this eyebrow-raising little app from Brett Slakin and > Brad Fitzpatrick: > > http://code.google.com/p/pubsubhubbub/ > > Jaiku folk seem to be involved too. > > Basically they have an experimental pub/sub system running on GAE, > using http instead of xmpp. The goals are somewhat broader than my > requirements - basically you have an atom feed at one url you post > data to, the atom feed updates a hub, and the hub then updates your > subscribers at different urls, so it's all a lot more distributed than > I need. > > I imagine one could do something similar rolled into one application > that handles publishing, the hub, and subscription (e.g. like jaiku or > twitter). But the big minus point that sticks out here for me in > pubsubhubub is that it completely avoids the question of the total > write-cost per update. It can afford to do that because subscribers > here are distributed all over the place on different domains and > servers etc. so the write load gets spread across all those different > urls/machines..it's not the concern of the hub running on appengine. > But if pubs/subs/hub are all rolled into one application on appengine, > then your application will be soaking up the total write cost. The hub > handles updating each subscriber in the background, I think, so it > does solve the issue of writing to n subscribers in the space of one > request..the publisher issues its update to the hub, returns > immediately, and then the hub partitions out the subscriber updates > and works on updating everyone over time. But it doesn't solve the > issue of monetary cost of doing n writes for n subscribers...if one > publisher had lots of subscribers (e.g. hundreds of thousands or > millions), each of their updates could cost $$$! > > But I guess one might be safe in assuming the average number of subs > per publisher will be much lower than that..and make the idea > financially feasible.. > > Anyway, I thought this project looked very interesting...it's giving > me a lot to chew on now.. > > On Mar 11, 3:50 pm, peterk <[email protected]> wrote: > > > Hmm. Good thinking Andrew. I've done a bit more digging along those > > lines..I don't think Jaiku has publically said if and what messaging > > protocols they might use, but there seems to be some speculation that > > it uses XMPP. > > > That might also neatly align with their comment that they are in some > > cases "extending" GAE in their port..and the recent news that XMPP is > > scheduled to come to GAE at some point in the next 6 months. Perhaps > > it's the work on the Jaiku port that's bringing that functionality... > > > I didn't know anything about messaging protocols or XMPP until I dug > > around after your post, so it's all new to me. I found a project > > purporting to provide XMPP functionality in advance of Google's > > official support, athttp://imified-demo.appspot.com/, but looking at > > the code for it, I cannot for the life of me see how this could be > > used to address the problems discussed in my original email. That > > project seems only to take instant messages via a POST to a URL and > > save them to the datastore..but there doesn't seem to be any of the > > 'instant distribution' of updates to people subscribed to your > > presence etc. that other XMPP software boast about. Using that xmpp > > project linked, there doesn't seem to be anything there about pushing > > updates to subscribed users etc. > > > I dug around the app gallery for other open source apps along similar > > lines..there are some would-be twitter/jaiku clones..all I've seen use > > non-messaging approaches, but they all make one compromise or another > > similar to solutions I'd come up with myself (e.g. taking the last 5 > > updates from friends and sorting them yourself in memory by > > date..which has its own issues, for example, if one user had made >5 > > updates recently..+ you'd inevitably hit a roadblock on all these > > reads beyond a certain number of friends..there's only so many friends > > you could do this for before it gets too slow). > > > Any thoughts/ideas...? > > > On Mar 11, 1:54 pm, Andrew Badera <[email protected]> wrote: > > > > you'd have to wonder if there's a push out to Gnip somewhere ... or if > > > protocol buffers are involved .. > > > > Thanks- > > > - Andy Badera > > > - [email protected] > > > - (518) 641-1280 > > > - Tech Valley Code Camp 2009.1:http://www.techvalleycodecamp.com/ > > > - Google me:http://www.google.com/search?q=andrew+badera > > > > On Wed, Mar 11, 2009 at 9:21 AM, peterk <[email protected]> wrote: > > > > > I just read on your blog (from January) the intention to release the > > > > appengine port of Jaiku as open source when the port is finished..but > > > > I was wondering if I could be so cheeky as to jump ahead with a couple > > > > of questions about it. The requirements of Jaiku seem to line up > > > > roughly similarly with issues I'm having in a slightly different > > > > context, that I'm finding pretty challenging to implement efficiently > > > > on app engine. > > > > > With your service, you seem to track updates for friends and other > > > > people I follow..so I might have a long list of people I'm following, > > > > and you feed me their updates. > > > > > How do you implement this on GAE? > > > > > I've been toying with a very similar problem for some time now. It > > > > seems to me you cannot chain together queries such as > > > > me.friends.updates.order(..) to get your friends' latest updates, for > > > > example. You can't make n writes to n update queues for n people > > > > following you, since writes are so costly. If I store my friends in a > > > > list of keys, this limits the number of friends I can query at a given > > > > time to 30. e.g. updates.all().filter('user IN', me.friends) is > > > > limited to 30 subqueries. I may have many more friends, so this > > > > approach doesn't seem to be sufficient. > > > > > I've been scratching my head over a similar problem for some time now, > > > > coming up with various hairbrained schemes that have been overly- > > > > complex, none of which deliver scalability to the nth degree. So I'd > > > > really, really, really appreciate any insight you could provide in > > > > implementing this kind of data model on GAE!! Many thanks! --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~----------~----~----~----~------~----~------~--~---
