Hi Arun, You don't say if you're using Python or Java. For Python debugging memory usage is hard at best; I'm not sure about Java.
In Python at least, every entity you load actually stores multiple copies of the data and it is decoded / encoded from protobufs many times. So the end result is that your memory usage will be higher than what you expect. Possibly by quite a bit depending on your data. It is also possible that you've got other issues in your code leading to memory leaks. That's harder to make general comments about though. Robert On Tue, Dec 27, 2011 at 11:58, Arun Shanker Prasad <[email protected]> wrote: > Hi All, > > Hope you had a wonderful holiday :) > > I am experiencing an issue for the past couple of weeks, am getting > soft memory exceeded errors and the instance is being terminated.. > This only started occuring recently. > > The pages experiencing this error are mostly reporting kind of pages, > i.e pages that deal with a number of records that are fetched based on > filters selected by the user and use django.core.paginator.Paginator > to paginate over the result. I have a maximum of 700 records that are > fetched for these reports, according to datastore statistics in the > admin console the average size of the entities being fetched are 4KB. > > Average Size of Entity: > 4 KBytes > > Some of the properties are reference properties and I do use prefetch > logic to load these references as well, but most of them are also > small entities. > > By my caclulations 158MB should be enough to handle a list of ~40448 > records! I am not able to figure out why the request is using this > much memory. > > I have tried eliminating most of the usual suspects (large string > concatination and so on), even tried deleting the large list after > they are processed and calling garbage collect, nothing seems to work. > Next step is to try and setup Apptrace for app engine. > > Can anyone out there help me with this? Anyone else have any similar > issues? How did you manage to sort it out? > > C 2011-12-27 10:03:54.089 > Exceeded soft private memory limit with 157.648 MB after servicing 4 > requests total > W 2011-12-27 10:03:54.090 > While handling this request, the process that handled this request was > found to be using too much memory and was terminated. This is likely > to cause a new process to be used for the next request to your > application. If you see this message frequently, you may have a memory > leak in your application. > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
