Yes,
I copy the response from support team ( Sorry but this was solved long time
ago, and i can't remember all the details )
-------
Getting back to your app configuration, the part which stands out the most
is this:
entrypoint: gunicorn 'client_account.wsgi:load_app("prod")'
It looks like you're using gunicorn with the default config, which is to
use one sync worker which limits your app to serving a single request at a
time. This could cause health check pings and other requests to time out if
the app is currently busy. One quick fix is to simply spawn more worker
processes to handle concurrent requests eg:
entrypoint: gunicorn -w 4 'client_account.wsgi:load_app("prod")'
---------
In summary , *gunicorn has to start with some workers* . By default,
gunicorn config launches only one . In case health_check request and a
common request arrives almost at the same time ,* gunicorn* cannot deal
with both of them. That's why 5xx HTTP status codes appear.
At the time of this issue , track those kind of issues in AppEngine logs
were a nightmare. Hope now is better.
Hope this answer can help you in some way ,
Cheers
El viernes, 3 de noviembre de 2017, 21:52:16 (UTC+1), Christian Aquino
escribió:
>
> Hi Jorge, I was wondering if you ever got to the bottom of this issue?
>
>
> Thanks,
> Christian
>
> On Friday, March 10, 2017 at 8:00:37 AM UTC-5, Jorge Barrachina wrote:
>>
>> Description of the problem :
>>
>> HTTP Cloud load Balancer 502 intermittent responses , doesn’t let
>> request reach our backend services. It seems that this problem rise up
>> randomly . Maybe related with
>> https://groups.google.com/forum/#!topic/google-appengine-downtime-notify/C_fCwHb73wc
>>
>> issued on February, 4th?
>>
>> Also, we were looking for alternative solutions and we’ve found this post
>> , that maybe has some correlation with this issue, and bring some light to
>> find what happened
>>
>>
>> https://blog.percy.io/tuning-nginx-behind-google-cloud-platform-http-s-load-balancer-305982ddb340#.rw4tbv6gl
>>
>>
>>
>> Issue date periods:
>>
>> March 9th, from 14:12:29.697 ( UTC/GMT +1 hour ) till March 10th,
>> 09:28:36.085 ( UTC/GMT +1 hour )
>>
>> Projects affected :
>>
>> oa-staging
>>
>> Services affected:
>>
>> clientaccount ( versions: release-2017-10-a , release-2017-8-a ,
>> release-2017-07-a)
>>
>>
>> Service url :
>>
>> https://clientaccount-dot-oa-staging.appspot-preview.com
>>
>>
>> Deployment Details of our stack :
>>
>>
>> app.yaml
>>
>> api_version: 1
>>
>> service: clientaccount
>>
>> runtime: python
>>
>> env: flex
>>
>> entrypoint: gunicorn 'client_account.wsgi:load_app("prod")'
>>
>> runtime_config:
>>
>> python_version: 3
>>
>> automatic_scaling:
>>
>> min_num_instances: 1
>>
>> max_num_instances: 5
>>
>> cool_down_period_sec: 120 # default value
>>
>> cpu_utilization:
>>
>> target_utilization: 0.8
>>
>>
>> requirements.txt
>>
>> bingads==v10.4.11
>>
>> boto3==1.3.0
>>
>> botocore==1.4.10
>>
>> docutils==0.12
>>
>> Flask==0.10.1
>>
>> Flask-Cors==2.1.2
>>
>> Flask-Security==1.7.5
>>
>> flask-swagger==0.2.12
>>
>> gcloud==0.13.0
>>
>> google-api-python-client==1.5.0
>>
>> googleads==5.0.0
>>
>> gunicorn==19.4.1
>>
>> itsdangerous==0.24
>>
>> Jinja2==2.8
>>
>> jmespath==0.9.0
>>
>> MarkupSafe==0.23
>>
>> mongoengine==0.10.6
>>
>> pymongo==3.2.2
>>
>> python-dateutil==2.5.2
>>
>> PyYAML==3.11
>>
>> recurly==2.4.2
>>
>> requests==2.9.1
>>
>> sendgrid==3.0.1
>>
>> six==1.10.0
>>
>> Werkzeug==0.11.4
>>
>> wheel==0.24.0
>>
>>
>> Transaction Example Details :
>>
>>
>> If you want to check the transaction on Google Cloud Console , here is
>> the link to check it out :
>>
>>
>>
>> https://console.cloud.google.com/logs/viewer?project=oa-staging&hl=es&minLogLevel=0&expandAll=false&resource=http_load_balancer&advancedFilter=resource.type%3D%22http_load_balancer%22%0Aresource.labels.zone%3D%22global%22%0Aresource.labels.project_id%3D%22oa-staging%22%0Atimestamp%3D%222017-03-09T15:35:43.842872262Z%22%0AinsertId%3D%221snolhbg22kclfq%22×tamp=2017-03-09T15:35:43.842872262Z
>>
>> GET
>> https://clientaccount-dot-oa-staging.appspot-preview.com/check_token/**********LONG_TOKEN*************
>>
>>
>> This request arrives to the HTTP Cloud Load Balancer at : 16:35:43.842
>> (Madrid Time)
>>
>>
>> Trace Log of the HTTP Cloud Balancer
>>
>> {
>>
>> insertId: "1snolhbg22kclfq"
>>
>> jsonPayload: {
>>
>> statusDetails: "failed_to_connect_to_backend"
>>
>> @type: "
>> type.googleapis.com/google.cloud.loadbalancing.type.LoadBalancerLogEntry"
>>
>>
>> }
>>
>> httpRequest: {
>>
>> requestMethod: "GET"
>>
>> requestUrl: "
>> https://clientaccount-dot-oa-staging.appspot-preview.com/check_token/*********LONG_TOKEN*************";
>>
>>
>>
>> requestSize: "1370"
>>
>> status: 502
>>
>> responseSize: "421"
>>
>> remoteIp: "34.197.229.75"
>>
>> }
>>
>> resource: {
>>
>> type: "http_load_balancer"
>>
>> labels: {
>>
>> url_map_name: ""
>>
>> forwarding_rule_name: ""
>>
>> backend_service_name: ""
>>
>> target_proxy_name: ""
>>
>> zone: "global"
>>
>> project_id: "oa-staging"
>>
>> }
>>
>> }
>>
>> timestamp: "2017-03-09T15:35:43.842872262Z"
>>
>> severity: "WARNING"
>>
>> logName: "projects/oa-staging/logs/requests"
>>
>> }
>>
>>
>> The connection with our backend (
>> https://clientaccount-dot-oa-staging.appspot-preview.com ) stopped . If
>> you look into the jsonPayload of the trace provided , it says
>> “failed_to_connect_to_backend” so it means that our backend seems to be
>> not processing the request arriving from the http load balancer.
>>
>> This is weird , because we didn’t change our stack and it was working
>> days before of the date reported.
>>
>> Could you tell us what happened , our provide some logs to review with
>> our team.
>>
>> Thanks for your support
>>
>>
--
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit
https://groups.google.com/d/msgid/google-appengine/2425dbfd-8e01-4183-9c6f-b16dea7bc9eb%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
