Happening in Ocata too: http://logs.openstack.org/86/474286/1/gate/gate-
tempest-dsvm-neutron-dvr-ubuntu-
xenial/2b247e0/logs/screen-q-svc.txt.gz#_2017-06-26_21_16_20_582
** Changed in: tempest
Importance: Medium => High
** Also affects: devstack
Importance: Undecided
Status: New
** Also affects: keystone
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Identity (keystone).
https://bugs.launchpad.net/bugs/1681866
Title:
Bad response code while validating token: 502
Status in devstack:
New
Status in OpenStack Identity (keystone):
New
Status in tempest:
Confirmed
Bug description:
Found this while investigating a gate failure [1].
Tempest logs say "2017-04-11 10:07:02,765 23082 INFO
[tempest.lib.common.rest_client] Request
(TestSecurityGroupsBasicOps:_run_cleanups): 503 DELETE
https://198.72.124.138:8774/v2.1/servers/f736a878-2ac4-4c37-b6a8-e5cd8df5a7fd
0.018s"
That 503 looks suspicious. So I go to the nova-api logs. Which gives
2017-04-11 10:07:02.762 32191 ERROR keystonemiddleware.auth_token [...] Bad
response code while validating token: 502
2017-04-11 10:07:02.763 32191 WARNING keystonemiddleware.auth_token [...]
Identity response: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>502 Proxy Error</title>
</head><body>
<h1>Proxy Error</h1>
<p>The proxy server received an invalid
response from an upstream server.<br />
The proxy server could not handle the request <em><a
href="/identity_admin/v3/auth/tokens">GET /identity_admin/v3/auth/tokens</a></em>.<p>
Reason: <strong>Error reading from remote server</strong></p></p>
<hr>
<address>Apache/2.4.18 (Ubuntu) Server at 198.72.124.138 Port 443</address>
</body></html>
2017-04-11 10:07:02.763 32191 CRITICAL keystonemiddleware.auth_token
[...] Unable to validate token: Failed to fetch token data from
identity server
So Apache is complaining, some network connection issue, related to
proxy-ing. So I open "logs/apache/tls-proxy_error.txt.gz" and find
[Tue Apr 11 10:07:02.761420 2017] [proxy_http:error] [pid 7136:tid
140090189690624] (20014)Internal error (specific information not available):
[client 198.72.124.138:38722] [frontend 198.72.124.138:443] AH01102: error
reading status line from remote server 198.72.124.138:80
[Tue Apr 11 10:07:02.761454 2017] [proxy:error] [pid 7136:tid
140090189690624] [client 198.72.124.138:38722] [frontend 198.72.124.138:443]
AH00898: Error reading from remote server returned by
/identity_admin/v3/auth/tokens
Interesting. Google says that adding "proxy-initial-not-pooled" to the
apache2 vhost config could help.
Anyway, a good elasticsearch query for this is
message:"Bad response code while validating token: 502"
8 hits, no worries.
[1] : http://logs.openstack.org/03/455303/2/check/gate-tempest-dsvm-
neutron-full-ubuntu-xenial/aa8c7fd/console.html
To manage notifications about this bug go to:
https://bugs.launchpad.net/devstack/+bug/1681866/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
