Today we started seeing our deployed services on AppEngine standard
environment fail with a lot of warnings saying "*Container terminated due
to sandbox error.*"
The code also shows a lot of 500 OPTIONS requests, which has the following
INFO log message
0: {
logMessage: "This request caused a new process to be started for your
application, and thus caused your application code to be loaded for the
first time. This request may thus take longer and use more CPU than a
typical request for your application."
severity: "INFO"
time: "2021-11-19T23:57:23.329869Z"
}
Splitting or migrating traffic to previous versions that we deployed and
were working before did not solve the problem, the service would start for
few minutes then run into the same issue. Since this is a Standard
Environment, we cannot stop the instances, instead we can re-deploy, and
when we did that the service started working but again only for few minutes
before it fails.
We suspected a recent change in our code that might have resulted in
throwing those 500 errors, and thought that these errors are causing
AppEngine to kill the instance (per docs it says that 20 consecutive 5xx
would make AppEngine kill the instance). We deployed this fix to both prod
and staging, and both worked, but after some time ONLY staging started
failing this time. Prod is still working even an hour later.
The only difference in configurations between staging and prod are:
1. Prod is class F4 while staging is F2
2. Prod has min_idle_instances set to 3 while staging set to 0
3. Prod has max_instances set to 5, while it's not set in staging
Would setting min_idle_instances in staging to something > 0 help fix this,
but even if, the question remains, why did things suddenly started failing.
Any help or ideas?
--
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/google-appengine/0af33f48-9f08-4cf6-8b9e-d9ba2948cf21n%40googlegroups.com.
