> At this point I think the GAE toolbox is empty. How about Cloud SQL?
On Wednesday, August 1, 2012 12:12:05 AM UTC+2, Jeff Schnitzer wrote: > > If Richard's metrics are right (he has created a very exact timing > system which accounts for skew, so I suspect they are), there's > sometimes a 4s+ delay between submitting a task to a pull queue and > when it becomes available for lease. This makes pull queues difficult > for precisely timing data collection and reaping in a 10s window. > > Honestly I'm struggling to find an in-appengine solution to this > problem. Let's say at T+0 clients submit 1000 data points. At T+10s, > the data must be collated to produce a leaderboard, and shortly > afterwards clients fetch the leaderboard. > > * Saving to the datastore and querying for the data points suffers > from eventual consistency issues. It works mostly but not always. > > * Putting items in a pull queue and then fetching them out in batch > would be the most logical solution, but the queue delay causes > problems. There doesn't seem to be any defined limit to queue > latency. > > * Using in-memory state in a Backend would be the next logical > solution, but backends can't handle the throughput. > > At this point I think the GAE toolbox is empty. It might be possible > to hack something together using memcache but it would be difficult > and fail whenever memcache resets. > > At this point all I can think of is to use something external to GAE > to queue the submissions. A simple custom in-memory server using > technology that handles concurrency well (eg Java) would work. For > something more off-the-shelf, a Redis instance fronted by a couple web > frontends to handle the submit() and reap() requests. At least we > *know* that Redis can handle thousands of submissions per second. > > Jeff > > On Tue, Jul 31, 2012 at 1:54 PM, Richard Watson > <[email protected]> wrote: > > How many queues are you using? Could you add more? > > Can you batch data into fewer tasks? If they're arriving at such a high > > rate, batching and adding fewer tasks might help. > > > > > > On Tuesday, July 31, 2012 10:10:12 PM UTC+2, Richard wrote: > >> > >> Yeah, but my worry is that if the queue architecture cannot handle 160 > >> entries... how will it handle > 1000 ? Remember these entries hang > around > >> for the next game round. So at that point, each client is also going > to > >> have to remove duplicates from other people that were left over from > the > >> previous round (hoping this makes sense!). Or else I purge the queue > in > >> between ... after attempting to patch the leaderboards a second time > with > >> the extra entries. > >> > >> It just seems like a hack on top of a hack and extremely ugly.... > > > > -- > > You received this message because you are subscribed to the Google > Groups > > "Google App Engine" group. > > To view this discussion on the web visit > > https://groups.google.com/d/msg/google-appengine/-/SFiRDIKnOAgJ. > > > > To post to this group, send email to [email protected]. > > To unsubscribe from this group, send email to > > [email protected]. > > For more options, visit this group at > > http://groups.google.com/group/google-appengine?hl=en. > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/jA0cQeGNs4IJ. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
