As I wrote earlier in the thread, the use case I'm thinking through is similar to yours: basically a caching proxy server. It's really not a matter of "writing good code." There is hardly any code to write.
I'm not complaining, exactly. I'm more just puzzled. I can see how GC could get behind, but in that case, any reasonable VM would just call time out for a bit and get things back in line. As you know, the bigger the available ram, the lower the bandwidth (inbound), and the lower the memcache & db ops. I'm surprised you, of all people, aren't bitching about the effective available ram being so much lower than the stated number. I recall that you implemented your app in Java. Was the situation better over there? -Joshua On Nov 25, 2011, at 2:19 PM, Brandon Wirtz wrote: > You get 128 soft limit, you start hitting soft limit at about 110 megs > depending on the requests per second. If your RPS gets very high Garbage > collection doesn’t keep up and you use more memory. Unless you are really > careful about how you use your variables lots of times you will end up with > more than one copy of things in memory. 128M is nothing. When you are > working with single threads it isn’t so bad, things flush with each request, > but when you have multiple threads everything is in memory at once. > > Quit complaining, if you write good code 128 will do you, just don’t try to > put your entire datastore in ram, that’s not what it is there for. > > From: [email protected] > [mailto:[email protected]] On Behalf Of Joshua Smith > Sent: Friday, November 25, 2011 11:01 AM > To: [email protected] > Subject: Re: [google-appengine] Using RAM instead of datastore - any limits? > > Good start. Now the other 67 megs? > > I'm no Python expert, but if it really takes 60+ megs to just load the > executing environment and code (which, seriously, is probably only a few K!), > then, well, I'm speechless... > > On Nov 25, 2011, at 12:07 PM, Brandon Wirtz wrote: > > > You have to remember that there is garbage collection, indexing, pointers, > all the variables you loaded, and all the imports. > > > From: [email protected] > [mailto:[email protected]] On Behalf Of Joshua Smith > Sent: Friday, November 25, 2011 5:43 AM > To: [email protected] > Subject: Re: [google-appengine] Using RAM instead of datastore - any limits? > > Wow - only 60? I thought python's interpreter was pretty efficient. Where's > the other half of the memory going? > > On Nov 24, 2011, at 3:02 PM, Brandon Wirtz wrote: > > > > I have local memory as part of my Caching solution which is a Reverse Caching > Proxy. You can’t have a very big site in Local memory because you only get > about 60 megs effective and memory isn’t shared between instances. > > From: [email protected] > [mailto:[email protected]] On Behalf Of Joshua Smith > Sent: Thursday, November 24, 2011 5:13 AM > To: [email protected] > Subject: Re: [google-appengine] Using RAM instead of datastore - any limits? > > The case I'm thinking of is a proxy server, so that isn't really an issue > here. > > On Nov 24, 2011, at 3:58 AM, Gerald Tan wrote: > > > > > Don't forget that caching your entities in instance memory will mean that it > will become stale if the entity is updated from another instance, and there > is no way of knowing that happens without querying the datastore. > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To view this discussion on the web visit > https://groups.google.com/d/msg/google-appengine/-/OfGkDVe2NhwJ. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
