Re: [appengine-java] Re: Slow cold starts
I would like to add my voice to this expression of concern that Google is not delivering on its promises. I have been running a GAEJ Saas since the end of 2009. On Friday I had my first discussion with a paying client of mine who now has serious doubts that my application will be able to support his business, given the performance he's seen in the last few weeks, notably the big delays seen on Friday. I was checking the system status that day and it said anomaly across all languages and investigating Peformance was truly terrible with errors, huge latencies and lots of loading requests. I was keen to see what the official investigation would turn up, but incredulous to see that its now just marked as a tick for that day! To me this is a serious breach of trust. It smells of cover up. I have a paid app, and have been experimenting for a while now with fine-tuning, min instances, max instances, pending latencies etc. I am now experimenting with warm-up requests but even that is not very evident, and I have yet to have it working. Documentation is definitely lacking in this crucial area. From what I read on these forums lately I see a huge amount of concern from the community and a deafening silence from Google. Is this really the thanks we get for being early adopters and trusting Google to deliver on their promises? Don't get me wrong I've been very impressed with GAE and the rate at which features are being rolled out, however this cannot be at the expense of stability and performance. I really fear I might now lose everything that I've built up over the last 18 months because my client trust is being eroded by all of these problems. I can't even blame Google coz they haven't even put out a statement. Googlers, how do I raise this to a production issue? Its make or break now for my application. On Saturday, March 10, 2012 9:23:29 AM UTC+1, andrew wrote: Yes. More votes for a Google response on this. Anyone with a premium account and better support want to take up the cause on all our behalf? Submit a defect report we can all star? Please post link to it here -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine-java/-/6HX2lHpopPYJ. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] Re: Slow cold starts
I noticed that nearly all the non-tick days on the status pages say investigating, even going back over a year! On Mar 20, 2012 7:21 AM, doright doug.stodd...@gmail.com wrote: I would like to add my voice to this expression of concern that Google is not delivering on its promises. I have been running a GAEJ Saas since the end of 2009. On Friday I had my first discussion with a paying client of mine who now has serious doubts that my application will be able to support his business, given the performance he's seen in the last few weeks, notably the big delays seen on Friday. I was checking the system status that day and it said anomaly across all languages and investigating Peformance was truly terrible with errors, huge latencies and lots of loading requests. I was keen to see what the official investigation would turn up, but incredulous to see that its now just marked as a tick for that day! To me this is a serious breach of trust. It smells of cover up. I have a paid app, and have been experimenting for a while now with fine-tuning, min instances, max instances, pending latencies etc. I am now experimenting with warm-up requests but even that is not very evident, and I have yet to have it working. Documentation is definitely lacking in this crucial area. From what I read on these forums lately I see a huge amount of concern from the community and a deafening silence from Google. Is this really the thanks we get for being early adopters and trusting Google to deliver on their promises? Don't get me wrong I've been very impressed with GAE and the rate at which features are being rolled out, however this cannot be at the expense of stability and performance. I really fear I might now lose everything that I've built up over the last 18 months because my client trust is being eroded by all of these problems. I can't even blame Google coz they haven't even put out a statement. Googlers, how do I raise this to a production issue? Its make or break now for my application. On Saturday, March 10, 2012 9:23:29 AM UTC+1, andrew wrote: Yes. More votes for a Google response on this. Anyone with a premium account and better support want to take up the cause on all our behalf? Submit a defect report we can all star? Please post link to it here -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine-java/-/6HX2lHpopPYJ. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] Re: Slow cold starts
Yes. More votes for a Google response on this. Anyone with a premium account and better support want to take up the cause on all our behalf? Submit a defect report we can all star? Please post link to it here -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine-java/-/kfFauAy3z08J. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] Re: Slow cold starts
That totally sucks. I can understand that it takes up more resources to have an app always loaded in an instance but when minimum instances is set, then it should mean that the app is kept in memory. If what you write is true (and I guess it is since nobody from Google has objected to it) then Google is not fulfilling what they themselves claim: Because App Engine keeps idle instances in reserve, it is unlikely that requests will enter the pending queue except in exceptionally high load spikes. http://code.google.com/appengine/docs/adminconsole/performancesettings.html On Thursday, March 8, 2012 10:10:03 PM UTC+1, Mark Rathwell wrote: Just wanted to follow up on this: Idle instances set to 1: 7 of 10 requests were loading requests (20-30 seconds) Idle instances set to 2: 3 of 9 requests were loading requests (and cost was about 4 times as much) This is a test app, with no other traffic, no new versions uploaded, etc., and these requests were about once a day for the last few weeks. I'm giving up on finding a setting that will mimic always on for low traffic apps, and just setting up a cron job to ping the apps regularly. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine-java/-/ly0xBCd9rVkJ. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] Re: Slow cold starts
Even we are finding these slow cold starts, very similar to what Jeff described earlier. Can someone from Google please respond? Thanks, Arun From: Anders Testson anders...@gmail.com To: google-appengine-java@googlegroups.com Sent: Friday, 9 March 2012 7:24 PM Subject: Re: [appengine-java] Re: Slow cold starts That totally sucks. I can understand that it takes up more resources to have an app always loaded in an instance but when minimum instances is set, then it should mean that the app is kept in memory. If what you write is true (and I guess it is since nobody from Google has objected to it) then Google is not fulfilling what they themselves claim: Because App Engine keeps idle instances in reserve, it is unlikely that requests will enter the pending queue except in exceptionally high load spikes. http://code.google.com/appengine/docs/adminconsole/performancesettings.html On Thursday, March 8, 2012 10:10:03 PM UTC+1, Mark Rathwell wrote: Just wanted to follow up on this: Idle instances set to 1: 7 of 10 requests were loading requests (20-30 seconds) Idle instances set to 2: 3 of 9 requests were loading requests (and cost was about 4 times as much) This is a test app, with no other traffic, no new versions uploaded, etc., and these requests were about once a day for the last few weeks. I'm giving up on finding a setting that will mimic always on for low traffic apps, and just setting up a cron job to ping the apps regularly. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine-java/-/ly0xBCd9rVkJ. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] Re: Slow cold starts
Just wanted to follow up on this: Idle instances set to 1: 7 of 10 requests were loading requests (20-30 seconds) Idle instances set to 2: 3 of 9 requests were loading requests (and cost was about 4 times as much) This is a test app, with no other traffic, no new versions uploaded, etc., and these requests were about once a day for the last few weeks. I'm giving up on finding a setting that will mimic always on for low traffic apps, and just setting up a cron job to ping the apps regularly. On Wed, Feb 29, 2012 at 10:51 PM, Anders blabl...@gmail.com wrote: Hmm... Not good. Google writes that setting minimum number of instances means keeping them in reserve so that cold starts will be unlikely unless there are exceptionally high load spikes. Sounds like a bug in GAE then if your application is set to minimum one instance and you still get many cold starts. Plus to Google: Seems like my email address is sometimes published because I usually get zero spam mail in my Gmail account except when I have recently posted something in a Google Group like this. So spammers seem to be able to somehow get my email address by scraping information from Google Groups or via some subscription mechanism. Very bad indeed if true (with the disclaimer that maybe it's not Google that is leaking email addresses). On Wednesday, February 29, 2012 7:24:45 PM UTC+1, Mark Rathwell wrote: Ok, interesting. Have you tried setting minimum instances to one? I have an app I've been testing, with billing enabled and minimum instances set to one. I check the app 1 - 2 times per day, and the overwhelming majority of those requests are cold starts. I just set the minimum instances to 2, so I'll see how it responds over the next few days. On Wed, Feb 29, 2012 at 6:25 AM, Anders wrote: Ok, interesting. Have you tried setting minimum instances to one? On Monday, February 27, 2012 8:40:41 AM UTC+1, Tapir wrote: From my experience, an app without billing enabled will encounter many many slow cold starts. That is why although my app visits is small but I still made my app billing enabled. But today, there are suddenly many cold starts again on my app even if it is billing enabled. On Feb 27, 12:30 am, Anders wrote: Thanks. The app no longer has billing enabled (needed to be able to adjust minimum instances), but that could be worth trying. On Sunday, February 26, 2012 2:26:00 PM UTC+1, Mike Lawrence wrote: I noticed similar behavior. I think the auto setting for minimum number of instances changed from two to zero active instances. I changed from auto to one, and my app is much more responsive. setting minimum instances higher will co$t you more On Feb 25, 9:36 am, Anders wrote: In the recent weeks the cold starts for Java apps have started to take a long time in my experience. More than three seconds to load a webpage is too long generally for a high quality end user experience. I assume it happens when the physical GAE servers become too crowded resulting in sluggish cold start loading of apps from the file system (and/or maybe because of delays in the distributed file system itself). -- -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine-java/-/GZ1p0GyAe6MJ. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
[appengine-java] Re: Slow cold starts
Ok, interesting. Have you tried setting minimum instances to one? On Monday, February 27, 2012 8:40:41 AM UTC+1, Tapir wrote: From my experience, an app without billing enabled will encounter many many slow cold starts. That is why although my app visits is small but I still made my app billing enabled. But today, there are suddenly many cold starts again on my app even if it is billing enabled. On Feb 27, 12:30 am, Anders blabl...@gmail.com wrote: Thanks. The app no longer has billing enabled (needed to be able to adjust minimum instances), but that could be worth trying. On Sunday, February 26, 2012 2:26:00 PM UTC+1, Mike Lawrence wrote: I noticed similar behavior. I think the auto setting for minimum number of instances changed from two to zero active instances. I changed from auto to one, and my app is much more responsive. setting minimum instances higher will co$t you more On Feb 25, 9:36 am, Anders blabl...@gmail.com wrote: In the recent weeks the cold starts for Java apps have started to take a long time in my experience. More than three seconds to load a webpage is too long generally for a high quality end user experience. I assume it happens when the physical GAE servers become too crowded resulting in sluggish cold start loading of apps from the file system (and/or maybe because of delays in the distributed file system itself). -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine-java/-/gCjF8DpwWJcJ. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] Re: Slow cold starts
Ok, interesting. Have you tried setting minimum instances to one? I have an app I've been testing, with billing enabled and minimum instances set to one. I check the app 1 - 2 times per day, and the overwhelming majority of those requests are cold starts. I just set the minimum instances to 2, so I'll see how it responds over the next few days. On Wed, Feb 29, 2012 at 6:25 AM, Anders blabl...@gmail.com wrote: Ok, interesting. Have you tried setting minimum instances to one? On Monday, February 27, 2012 8:40:41 AM UTC+1, Tapir wrote: From my experience, an app without billing enabled will encounter many many slow cold starts. That is why although my app visits is small but I still made my app billing enabled. But today, there are suddenly many cold starts again on my app even if it is billing enabled. On Feb 27, 12:30 am, Anders blabl...@gmail.com wrote: Thanks. The app no longer has billing enabled (needed to be able to adjust minimum instances), but that could be worth trying. On Sunday, February 26, 2012 2:26:00 PM UTC+1, Mike Lawrence wrote: I noticed similar behavior. I think the auto setting for minimum number of instances changed from two to zero active instances. I changed from auto to one, and my app is much more responsive. setting minimum instances higher will co$t you more On Feb 25, 9:36 am, Anders blabl...@gmail.com wrote: In the recent weeks the cold starts for Java apps have started to take a long time in my experience. More than three seconds to load a webpage is too long generally for a high quality end user experience. I assume it happens when the physical GAE servers become too crowded resulting in sluggish cold start loading of apps from the file system (and/or maybe because of delays in the distributed file system itself). -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine-java/-/gCjF8DpwWJcJ. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] Re: Slow cold starts
Hmm... Not good. Google writes that setting minimum number of instances means keeping them in reserve so that cold starts will be unlikely unless there are exceptionally high load spikes. Sounds like a bug in GAE then if your application is set to minimum one instance and you still get many cold starts. Plus to Google: Seems like my email address is sometimes published because I usually get zero spam mail in my Gmail account except when I have recently posted something in a Google Group like this. So spammers seem to be able to somehow get my email address by scraping information from Google Groups or via some subscription mechanism. Very bad indeed if true (with the disclaimer that maybe it's not Google that is leaking email addresses). On Wednesday, February 29, 2012 7:24:45 PM UTC+1, Mark Rathwell wrote: Ok, interesting. Have you tried setting minimum instances to one? I have an app I've been testing, with billing enabled and minimum instances set to one. I check the app 1 - 2 times per day, and the overwhelming majority of those requests are cold starts. I just set the minimum instances to 2, so I'll see how it responds over the next few days. On Wed, Feb 29, 2012 at 6:25 AM, Anders wrote: Ok, interesting. Have you tried setting minimum instances to one? On Monday, February 27, 2012 8:40:41 AM UTC+1, Tapir wrote: From my experience, an app without billing enabled will encounter many many slow cold starts. That is why although my app visits is small but I still made my app billing enabled. But today, there are suddenly many cold starts again on my app even if it is billing enabled. On Feb 27, 12:30 am, Anders wrote: Thanks. The app no longer has billing enabled (needed to be able to adjust minimum instances), but that could be worth trying. On Sunday, February 26, 2012 2:26:00 PM UTC+1, Mike Lawrence wrote: I noticed similar behavior. I think the auto setting for minimum number of instances changed from two to zero active instances. I changed from auto to one, and my app is much more responsive. setting minimum instances higher will co$t you more On Feb 25, 9:36 am, Anders wrote: In the recent weeks the cold starts for Java apps have started to take a long time in my experience. More than three seconds to load a webpage is too long generally for a high quality end user experience. I assume it happens when the physical GAE servers become too crowded resulting in sluggish cold start loading of apps from the file system (and/or maybe because of delays in the distributed file system itself). -- -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine-java/-/GZ1p0GyAe6MJ. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] Re: Slow cold starts
I see now that my email address indeed is visible. How is that possible?! Has Google not been able to fix such a serious privacy leak yet?!!! -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine-java/-/NjhNGFhzc60J. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
[appengine-java] Re: Slow cold starts
I noticed similar behavior. I think the auto setting for minimum number of instances changed from two to zero active instances. I changed from auto to one, and my app is much more responsive. setting minimum instances higher will co$t you more On Feb 25, 9:36 am, Anders blabl...@gmail.com wrote: In the recent weeks the cold starts for Java apps have started to take a long time in my experience. More than three seconds to load a webpage is too long generally for a high quality end user experience. I assume it happens when the physical GAE servers become too crowded resulting in sluggish cold start loading of apps from the file system (and/or maybe because of delays in the distributed file system itself). -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
[appengine-java] Re: Slow cold starts
Thanks. The app no longer has billing enabled (needed to be able to adjust minimum instances), but that could be worth trying. On Sunday, February 26, 2012 2:26:00 PM UTC+1, Mike Lawrence wrote: I noticed similar behavior. I think the auto setting for minimum number of instances changed from two to zero active instances. I changed from auto to one, and my app is much more responsive. setting minimum instances higher will co$t you more On Feb 25, 9:36 am, Anders blabl...@gmail.com wrote: In the recent weeks the cold starts for Java apps have started to take a long time in my experience. More than three seconds to load a webpage is too long generally for a high quality end user experience. I assume it happens when the physical GAE servers become too crowded resulting in sluggish cold start loading of apps from the file system (and/or maybe because of delays in the distributed file system itself). -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine-java/-/7bbVZyuD5oAJ. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
[appengine-java] Re: Slow cold starts
From my experience, an app without billing enabled will encounter many many slow cold starts. That is why although my app visits is small but I still made my app billing enabled. But today, there are suddenly many cold starts again on my app even if it is billing enabled. On Feb 27, 12:30 am, Anders blabl...@gmail.com wrote: Thanks. The app no longer has billing enabled (needed to be able to adjust minimum instances), but that could be worth trying. On Sunday, February 26, 2012 2:26:00 PM UTC+1, Mike Lawrence wrote: I noticed similar behavior. I think the auto setting for minimum number of instances changed from two to zero active instances. I changed from auto to one, and my app is much more responsive. setting minimum instances higher will co$t you more On Feb 25, 9:36 am, Anders blabl...@gmail.com wrote: In the recent weeks the cold starts for Java apps have started to take a long time in my experience. More than three seconds to load a webpage is too long generally for a high quality end user experience. I assume it happens when the physical GAE servers become too crowded resulting in sluggish cold start loading of apps from the file system (and/or maybe because of delays in the distributed file system itself). -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
