Maybe you can use the client side of the process. Write a piece of Javascript that uses Ajax calls to perform multiple requests simultaneous (Ajax=asynchronous) that each retrieve part of the objects (there should be some range on the 'the_order' field, chop it in pieces). Return XML or Jason for the objects found. Combine the results of these queries and construct the webpage client side or by sending the whole bunch to the server in another Ajax call.
2010/5/8 Adrian Holovaty <[email protected]>: > Hi all, > > I've got an App Engine application in which I'm selecting hundreds of > entities from the same model, possibly even 1000 or 2000 of them -- > and I need all of them at the same time, for display in the browser. > (There is no way around this requirement; I need all of them at the > same time.) The Python code looks something like this, with model/ > property names changed: > > MyModel.all().filter('somekey =', x).order('the_order') > > This becomes unbearably slow when selecting hundreds or thousands of > records. I've read through the docs and various articles explaining > *why* that happens, but I haven't found any good advice on how to > solve it, aside from "put it in memcache" or "don't do that." I am > indeed using memcache in the app's view code, like so... > > if in_the_cache: > return cached > else: > result = MyModel.all().filter('somekey =', > x).order('the_order') > put_in_cache(result) > return result > > ...and I invalidate the cache whenever a write is made, but the values > change often enough that the caching is essentially useless (this is a > write-heavy app). > > One option I'm considering is to recalculate and cache the result > whenever a write happens, but that could result in a lot of > unnecessary CPU usage, because, again, this is a write-heavy app, and > I only do about one read for every 20-30 writes. > > Ideally there would be a way to optimize the way I model the data or > query the data, such that I could get the hundreds of objects in a > shorter period of time without sacrificing CPU usage. Are there any > fancy data modeling tricks I can use, like sharding them over multiple > models (which doesn't seem like it would make any difference), or > using ListProperty (in which case I might run into the limit of items > in a ListProperty, because each MyModel record has four attributes I > need to include), or something else...? > > Adrian > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
