-So are textProperties more efficient than StringProperties because they're not indexed?
You'd have to find the talk from Google IO to be sure. I believe it was the one about scalability, in the QA section. But yes, that is my understanding. -I noticed something weird: I do not have an index listed in appengine dashboard for the thumb datastore. Maybe that's what's causing this problem? To find all the images for a particular item I use back references. I don't know for sure how the indices work in relation to keys (which is what references are stored as), but you may need a manual index. My understanding is however that the datastore can't do queries at all if they're not indexed, so rather than being slow it wouldn't work at all. I'm sketchy on this though. Hopefully someone with more solid knowledge of the datastore can fill in the gaps here. -I see your point about Memcache & Cache handlers which should help optimize the system. But Isn't it important to get the basic functionality straightened out before using those extra tools? This isn't a very complex operation AT ALL; which is why it is so puzzling to me that it takes so many resources. There is a point to that, but designing from the ground up to be scalable and using caching wherever it makes sense, will save you a lot of headaches later on. It will likely also improve your code as you'll be building it more robustly from the get go. -Wouldn't adding etag--while increasing efficiency if I have the same users loading the same image again and again--actually decrease efficiency for users who are opening up an thumbnail for the first time? In that situation, I'd have another column for etags in my datastore being requested w/ every query. Yes, you absolutely should generate the etag when you save the thumbnail, and save it in the model itself. Caching it separately is however still desirable as you can then avoid pulling the rest of the data into memory if it's not needed, or you can opt to not cache the rest of the data at all, instead only caching the etag, to be more cache friendly. -Could this problem be a result of having two reference properties in one Model? Also, is back referencing inefficient? Should I instead just store all the Keys as strings & then query the datastore for all entries that have that Key? References are simply id's, and I know they are never queried unless you actually access them (e.g. entity.created_by.username), so there shouldn't be any overhead there that I can think of. I want to stress that this is just based on my personal experience with the system and what I've gleaned from talks. But I hope some of it helps you at least. On Sep 25, 10:50 am, iceanfire <[EMAIL PROTECTED]> wrote: > Hey Thomas, > > Thanks for the reply. > > -So are textProperties more efficient than StringProperties because > they're not indexed? > > -I noticed something weird: I do not have an index listed in appengine > dashboard for the thumb datastore. Maybe that's what's causing this > problem? To find all the images for a particular item I use back > references. > > -I see your point about Memcache & Cache handlers which should help > optimize the system. But Isn't it important to get the basic > functionality straightened out before using those extra tools? This > isn't a very complex operation AT ALL; which is why it is so puzzling > to me that it takes so many resources. > > -Wouldn't adding etag--while increasing efficiency if I have the same > users loading the same image again and again--actually decrease > efficiency for users who are opening up an thumbnail for the first > time? In that situation, I'd have another column for etags in my > datastore being requested w/ every query. > > -Could this problem be a result of having two reference properties in > one Model? Also, is back referencing inefficient? Should I instead > just store all the Keys as strings & then query the datastore for all > entries that have that Key? > > Thanks! > > On Sep 25, 1:52 am, Thomas Johansson <[EMAIL PROTECTED]> wrote: > > > At Google IO I believe it was mentioned that TextProperty's are fine > > for anything that you don't want indexed; They are no less efficient > > than strings for short data. You might want to look into trying that > > out with mime and type if you don't filter/order by them. > > > Other than that, I'd suggest trying to memcache the images, something > > to the effect of: > > > id = int(id) > > cache_key = 'ImageThumb:%i' % id > > image = memcache.get(cache_key) > > if not image: > > image = ImageThumb.get_by_id(id) > > if image: > > memcache.set(cache_key, image, 3600) > > if image: > > self.response.headers['Content-Type'] = str(image.mime) > > self.response.out.write(image.thumb) > > else: > > self.error(404) > > > In addition to that, I'd recommend you generate cache headers, in > > particular etag and expiration. A good idea would be to generate the > > etag when the ImageThumb is created, and then compare against that > > from cache, before you write out the image. As an added bonus you > > could store the etag on it's own in the cache, separate from the > > image, and only pull out the actual image in case the etags don't > > match. > > > On Sep 25, 4:59 am, iceanfire <[EMAIL PROTECTED]> wrote: > > > > I'm still having trouble with high-cpu warnings for thumbnails. This > > > time around I took some profiler data to see what's causing it. But > > > before I get into that here is the model i'm using: > > > > class ImageThumb(db.Model): > > > binId = db.IntegerProperty() > > > thumb = db.BlobProperty(default=None) > > > building = db.ReferenceProperty(Buildings) > > > apartment = db.ReferenceProperty(Apartments) > > > mime = db.StringProperty() > > > type = db.StringProperty(choices=['Floor Plan','Picture']) > > > created_by = db.UserProperty() > > > > So here's the Profiler cpu data: > > > 2538 function calls (2472 primitive calls) in 0.028 CPU seconds (rest > > > of the data:http://docs.google.com/Doc?id=dgfxff5_30hc9tjsgg) > > > > But here's the warning: This request used a high amount of CPU, and > > > was roughly 1.3 times over the average request CPU limit. High CPU > > > requests have a small quota, and if you exceed this quota, your app > > > will be temporarily disabled. > > > > Here's the megacycle info from the new Admin console: 1339mcycles 7kb > > > > Here's the code that pulls up the data for thumbnails: > > > class Apt_thumb (webapp.RequestHandler): > > > def get(self,id): > > > image = ImageThumb.get_by_id(int(id)) > > > if image: > > > self.response.headers['Content-Type'] = > > > str(image.mime) > > > self.response.out.write(image.thumb) > > > else: > > > self.error(404) > > > > The images stored are 4kb each in the datastore. As I said, I had this > > > problem earlier, so I had taken out the full image and put that in its > > > own Model. Clearly that didn't help. > > > > Any idea what's causing this high-cpu error & how I can fix it? > > > > Thanks! --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~----------~----~----~----~------~----~------~--~---
