Hi Jason, Thanks for the detailed answer. I'm very surprised that no one else is talking about these issues.
I'm using ndb and my appstats are off. I could see a incredible great improvement in my app when I turned off the stats. So, I recommend people only use stats for testing or debug. As you mentioned I'll try projection queries. Could you explain something more about NDB in-memory cache? Thanks again. Saludos. Moisés Belchín. 2013/10/12 Jason Galea <ja...@lecstor.com> > Hi Moises, > > we're currently trying to deal with this issue too. Not in mapreduce, just > regular handlers. > > same here - "Python 2.7 and F2 instances with 256MB" > > Early on I found that fetching 1000 entities and looping through them to > update a property would blow the instance. Reducing this to say 100 fixed > the issue. (than when create another task do do the next 100) > > "After handling this request" - as far as I understand this isn't so bad > unless you're blowing it on every request and starting a new instance is > detrimental. > > "While handling this request" - this concerns us most as the request does > not complete and breaks stuff.. and we see far too many of them. > > I've spent more than a little time trying to work out what is causing the > blowouts but as far as I've been able to work out, memory usage and what > causes it is near impossible (or just very, very hard). > > Are you using NDB? if so.. > - you could try disabling the in-memory cache. As I see it, even though > you only access one entity at a time, NDB's in memory cache will store them > all until the request is completed. > - you could try projection queries if you don't need the complete object > (or possibly even if you do). Projection queries get their data from an > index and the entities returned cannot be put() so I assume they are not > cached at all. We're trialling some fixes with these atm. > > ** If anyone knows any of this is incorrect, please let me know.. > > I'm actually surprised there is not more discussion of these issues from > what we have experienced so maybe we're doing something fundamentally > wrong, but I don't believe so. > > oh, is appstats turned on? I believe the most noticeable improvement we've > seen was when we turned it off.. > > regards, > > Jason > > > > > > > On Fri, Oct 11, 2013 at 5:51 PM, Moises Belchin > <moisesbelc...@gmail.com>wrote: > >> Hi Vinny, >> >> Thanks for the tips, but actually I'm not loading a file. I'm only using >> mapreduce lib for read all the entities for one of my kinds, work with them >> (I only read some properties to compose the csv line format) and then I >> write to CSV file on cloud storage using mapreduce FileOutputWriter. >> >> Any idea why I'm getting this Criticals memory errors? >> >> Thanks all again. >> >> >> Saludos. >> Moisés Belchín. >> >> >> 2013/10/10 Vinny P <vinny...@gmail.com> >> >>> On Thu, Oct 10, 2013 at 5:47 AM, Moises Belchin <moisesbelc...@gmail.com >>> > wrote: >>> >>>> I'm getting a lot of memory limit critical errors in my app when I use >>>> mapreduce library. >>>> I'm working with Python 2.7, app engine 1.8.5 and F2 instances with >>>> 256MB. >>>> >>>> Has anyone got these critical errors? >>>> >>>> >>> >>> Hello Moises, >>> >>> How large is your input file? Are you loading the entire file into >>> memory at once? If so, try moving to a BlobstoreLineInputReader which reads >>> in one line at a time - it reduces the memory being used during file >>> processing. >>> >>> >>> ----------------- >>> -Vinny P >>> Technology & Media Advisor >>> Chicago, IL >>> >>> App Engine Code Samples: http://www.learntogoogleit.com >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Google App Engine" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to google-appengine+unsubscr...@googlegroups.com. >>> To post to this group, send email to google-appengine@googlegroups.com. >>> Visit this group at http://groups.google.com/group/google-appengine. >>> For more options, visit https://groups.google.com/groups/opt_out. >>> >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Google App Engine" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to google-appengine+unsubscr...@googlegroups.com. >> To post to this group, send email to google-appengine@googlegroups.com. >> Visit this group at http://groups.google.com/group/google-appengine. >> For more options, visit https://groups.google.com/groups/opt_out. >> > > > > -- > Jason Galea > lecstor.com > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to google-appengine+unsubscr...@googlegroups.com. > To post to this group, send email to google-appengine@googlegroups.com. > Visit this group at http://groups.google.com/group/google-appengine. > For more options, visit https://groups.google.com/groups/opt_out. > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine. For more options, visit https://groups.google.com/groups/opt_out.
