Just a head's up - Jaiku has gone open source :) http://code.google.com/p/jaikuengine/
At a very brief first glance, I see references to xmpp stuff and more..going to try and map out the code and see what goodies might be there, could be stuff of interest beyond pub/sub too. On Mar 13, 1:28 pm, peterk <[email protected]> wrote: > Unfortunately I do need to query them based on subscriber_id..so I > can't pack them into a non-indexed property. > > Retrieving updates particular user has subscribed to is blazingly fast > though...that's the gain in the end, I can query and fetch 1000 > updates for a user sorted by date in 20-30ms-cpu. Love that :p In my > hacky approaches previously where I tried to write once and then > 'gather', I had to do lots of in-memory sorting and stuff, and even > the results often wouldn't be totally accurate. > > I'm going to keep toying with the write end of things though..because > in my full app, I may need to do write to other entities along with > subscribers to do certain things I'm trying to achieve. So I'm going > to be looking for every opportunity possible to optimise the cost of > an 'update', which in my case may go beyond notifying subscribers. So > any thoughts/ideas on further optimisation are more than welcome!! > > @Paul > > If you've more subscribers than will fit in one 'group' you'll need > multiple groups, correct. So you'll have n writes, where n = number of > subscribers/group-size, rounded up to the nearest whole number. Even > with the costly index creation for each of these 'group' entities > though, it should still work out a fair bit cheaper than writing a > seperate entity for each subscriber. > > On Mar 13, 11:47 am, bFlood <[email protected]> wrote: > > > @peterk - if you don't need to query by the subscriber, you could > > alternatively pack the list of subscribers for a feed into a > > TextProperty so it is not indexed. I use TextProperty a lot to store > > large lists of geometry data and they work out pretty well > > > @brett - async! looking forward to it in future GAE builds. thanks > > > cheers > > brian > > > On Mar 13, 5:37 am, peterk <[email protected]> wrote: > > > > I was just toying around with this idea yesterday Brett.. :D I did > > > some profiling, and it would reduce the write cost per subscriber to > > > about 24ms-40ms (depending on the number of subscribers you have..more > > > = lower cost per avg), from 100-150ms. These are rough numbers with > > > entities I was using, I have to do some more accurate profiling.. > > > > When I first thought about doing this, I was thinking ":o I'll reduce > > > write cost by a factor of hundreds!", but as it turns out, the extra > > > index update time for an entity with a large number of list property > > > entries eats into that saving significantly. > > > > But it still is a saving. Funnily enough the per subscriber saving > > > increases (to a point) the more subscribers you have. > > > > I'm not sure if there's anything one can do to optimise index creation > > > time with large lists.. I'm going to do some more work as well to see > > > if there's an optimum 'batch size' for grouping subscribers > > > together..at first blush, as mentioned above, it seems the larger the > > > better (up to the per entity property/index cap of course). > > > > Thanks also for the insight on pubsubhubub..I eagerly await updates on > > > that front :) Thank you!! > > > > On Mar 13, 8:05 am, Paul Kinlan <[email protected]> wrote: > > > > > Just Curious, > > > > > For other pub/sub-style systems where you want to write to the > > > > Datastore, the trick is to use list properties to track the > > > > subscribers you've published to. So for instance, instead of writing a > > > > single entity per subscriber, you write one entity with 1000-2000 > > > > subscriber IDs in a list. Then all queries for that list with an > > > > equals filter for the subscriber will show the entity. This lets you > > > > pack a lot of information into a single entity write, thus minimizing > > > > Datastore overhead, cost, etc. Does that make sense? > > > > > So if you have over the 5000 limit in the subscribers would you write > > > > the > > > > entity twice? Each with differnt subscriber id's? > > > > > Paul > > > > > 2009/3/13 Brett Slatkin <[email protected]> > > > > > > Heyo, > > > > > > Good finds, peterk! > > > > > > pubsubhubbub uses some of the same techniques thatJaikuuses for > > > > > doing one-to-many fan-out of status message updates. The migration is > > > > > underway as we speak > > > > > (http://www.jaiku.com/blog/2009/03/11/upcoming-service-break/). I > > > > > believe the code should be available very soon. > > > > > > 2009/3/11 peterk <[email protected]>: > > > > > > > The app is actually live here: > > > > > > >http://pubsubhubbub.appspot.com/ > > > > > >http://pubsubhubbub-subscriber.appspot.com/ > > > > > > > (pubsubhubbub-publisher isn't there, but it's trivial to upload your > > > > > > own.) > > > > > > > This suggests it's working on appengine as it is now. Been looking > > > > > > through the source, and I'm not entirely clear on how the > > > > > > 'background > > > > > > workers' are actually working..there are two, one for pulling > > > > > > updates > > > > > > to feeds from publishers, and one for propogating updates to > > > > > > subscribers in batches. > > > > > > > But like I say, I can't see how they're actually started and running > > > > > > constantly. There is a video here of a live demonstration: > > > > > > >http://www.veodia.com/player.php?vid=fCNU1qQ1oSs > > > > > > > The background workers seem to be behaving as desired there, but I'm > > > > > > not sure if they were just constantly polling some urls to keep the > > > > > > workers live for the purposes of that demo, or if they're actually > > > > > > running somehow constantly on their own.. I can't actually get the > > > > > > live app at the urls above to work, but not sure if it's because > > > > > > background workers aren't really working, or because i'm feeding it > > > > > > incorrect urls/configuration etc. > > > > > > Ah sorry yeah I still have the old version of the source running on > > > > > pubsubhubbub.appspot.com; I need to update that with a more recent > > > > > build. Sorry for the trouble! It's still not quite ready for > > > > > widespread use, but it should be soon. > > > > > > The way pubsubhubbub does fan-out, there's no need to write an entity > > > > > for each subscriber of a feed. Instead, each time it consumes a task > > > > > from the work queue it will update the current iterator position in > > > > > the query result of subscribers for a URL. Subsequent work requests > > > > > will offset into the subscribers starting at the iterator position. > > > > > This works well in this case because it's using urlfetch to actually > > > > > notify subscribers, instead of writing to the Datastore. > > > > > > For other pub/sub-style systems where you want to write to the > > > > > Datastore, the trick is to use list properties to track the > > > > > subscribers you've published to. So for instance, instead of writing a > > > > > single entity per subscriber, you write one entity with 1000-2000 > > > > > subscriber IDs in a list. Then all queries for that list with an > > > > > equals filter for the subscriber will show the entity. This lets you > > > > > pack a lot of information into a single entity write, thus minimizing > > > > > Datastore overhead, cost, etc. Does that make sense? > > > > > > @bFlood: Indeed, the async_apiproxy.py code is interesting. Not much > > > > > to say about that at this time, besides the fact that it works. =) > > > > > > -Brett --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~----------~----~----~----~------~----~------~--~---
