Hi all,
I've got an App Engine application in which I'm selecting hundreds of
entities from the same model, possibly even 1000 or 2000 of them --
and I need all of them at the same time, for display in the browser.
(There is no way around this requirement; I need all of them at the
same time.) The Python code looks something like this, with model/
property names changed:
MyModel.all().filter('somekey =', x).order('the_order')
This becomes unbearably slow when selecting hundreds or thousands of
records. I've read through the docs and various articles explaining
*why* that happens, but I haven't found any good advice on how to
solve it, aside from "put it in memcache" or "don't do that." I am
indeed using memcache in the app's view code, like so...
if in_the_cache:
return cached
else:
result = MyModel.all().filter('somekey =',
x).order('the_order')
put_in_cache(result)
return result
...and I invalidate the cache whenever a write is made, but the values
change often enough that the caching is essentially useless (this is a
write-heavy app).
One option I'm considering is to recalculate and cache the result
whenever a write happens, but that could result in a lot of
unnecessary CPU usage, because, again, this is a write-heavy app, and
I only do about one read for every 20-30 writes.
Ideally there would be a way to optimize the way I model the data or
query the data, such that I could get the hundreds of objects in a
shorter period of time without sacrificing CPU usage. Are there any
fancy data modeling tricks I can use, like sharding them over multiple
models (which doesn't seem like it would make any difference), or
using ListProperty (in which case I might run into the limit of items
in a ListProperty, because each MyModel record has four attributes I
need to include), or something else...?
Adrian
--
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To post to this group, send email to [email protected].
To unsubscribe from this group, send email to
[email protected].
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.
