Public bug reported: Noticed this while working on something else, if the API is going to return a non 2xx HTTP success status code, a lot of request details are logged including the user's unsanitized auth token. In the past, operators considered this to be a security issue despite logging only at level DEBUG. For this reason I am opening a bug for review.
This particular logging code was added in the Zed release: https://review.opendev.org/c/openstack/nova/+/806683 These are logged a lot when using OSC + server names because OSC always tries to lookup a name as a UUID (which will fail with 404) before it falls back on trying it as an ID. So commands such as 'openstack server show MyServer' will produce debug logs like the following. Example log for GET /servers HTTP 404: Mar 28 01:11:57 ubuntu-focal [email protected][3890335]: INFO nova.api.openstack.wsgi [None req-f5dbcb16-318b-4b19-96a6-62492ac94677 demo demo] HTTP exception thrown: Instance test could not be found. Mar 28 01:11:57 ubuntu-focal [email protected][3890335]: DEBUG nova.api.openstack.wsgi [None req-f5dbcb16-318b-4b19-96a6-62492ac94677 demo demo] Request method failure captured: Mar 28 01:11:57 ubuntu-focal [email protected][3890335]: request: GET /compute/v2.1/servers/test HTTP/1.1 Mar 28 01:11:57 ubuntu-focal [email protected][3890335]: Accept: application/json Mar 28 01:11:57 ubuntu-focal [email protected][3890335]: Accept-Encoding: gzip, deflate Mar 28 01:11:57 ubuntu-focal [email protected][3890335]: Connection: keep-alive Mar 28 01:11:57 ubuntu-focal [email protected][3890335]: Content-Length: 0 Mar 28 01:11:57 ubuntu-focal [email protected][3890335]: Host: 192.168.44.11 Mar 28 01:11:57 ubuntu-focal [email protected][3890335]: Openstack-System-Scope: None Mar 28 01:11:57 ubuntu-focal [email protected][3890335]: User-Agent: python-novaclient Mar 28 01:11:57 ubuntu-focal [email protected][3890335]: X-Auth-Token: gAAAAABkIj7cGDJYKbnQalHot3qfNuAY1AMwMdKFP0kVQB6_8HRCSizDQpxMfspjx2S7t8rMWPSwYwg0-Yox2QM9E3KPWVq72YPl-cr8XwlDIn-ev9WjpmkmCtlOqOs0M0rOSvQggxdNB0xLx 2-gYiWUyMAYW6A1vXcup7Rvs8-YFetPKr2vGnY [...] Full log trace: https://paste.openstack.org/show/bx9CmbgOsDNrIn16PfUW ** Affects: nova Importance: Undecided Status: New ** Tags: api security ** Description changed: Noticed this while working on something else, if the API is going to return a non 2xx HTTP success status code, a lot of request details are logged including the user's unsanitized auth token. In the past, operators considered this to be a security issue despite logging only at - level DEBUG. For this reason I am opening a bug for review. + level DEBUG. For this reason, I am opening a bug for review. Example for GET /servers HTTP 404: Mar 28 01:11:57 ubuntu-focal [email protected][3890335]: INFO nova.api.openstack.wsgi [None req-f5dbcb16-318b-4b19-96a6-62492ac94677 demo demo] HTTP exception thrown: Instance test could not be found. Mar 28 01:11:57 ubuntu-focal [email protected][3890335]: DEBUG nova.api.openstack.wsgi [None req-f5dbcb16-318b-4b19-96a6-62492ac94677 demo demo] Request method failure captured: Mar 28 01:11:57 ubuntu-focal [email protected][3890335]: request: GET /compute/v2.1/servers/test HTTP/1.1 Mar 28 01:11:57 ubuntu-focal [email protected][3890335]: Accept: application/json Mar 28 01:11:57 ubuntu-focal [email protected][3890335]: Accept-Encoding: gzip, deflate Mar 28 01:11:57 ubuntu-focal [email protected][3890335]: Connection: keep-alive Mar 28 01:11:57 ubuntu-focal [email protected][3890335]: Content-Length: 0 Mar 28 01:11:57 ubuntu-focal [email protected][3890335]: Host: 192.168.44.11 Mar 28 01:11:57 ubuntu-focal [email protected][3890335]: Openstack-System-Scope: None Mar 28 01:11:57 ubuntu-focal [email protected][3890335]: User-Agent: python-novaclient Mar 28 01:11:57 ubuntu-focal [email protected][3890335]: X-Auth-Token: gAAAAABkIj7cGDJYKbnQalHot3qfNuAY1AMwMdKFP0kVQB6_8HRCSizDQpxMfspjx2S7t8rMWPSwYwg0-Yox2QM9E3KPWVq72YPl-cr8XwlDIn-ev9WjpmkmCtlOqOs0M0rOSvQggxdNB0xLx 2-gYiWUyMAYW6A1vXcup7Rvs8-YFetPKr2vGnY [...] Full log trace: https://paste.openstack.org/show/bx9CmbgOsDNrIn16PfUW ** Description changed: Noticed this while working on something else, if the API is going to return a non 2xx HTTP success status code, a lot of request details are logged including the user's unsanitized auth token. In the past, operators considered this to be a security issue despite logging only at - level DEBUG. For this reason, I am opening a bug for review. + level DEBUG. For this reason I am opening a bug for review. - Example for GET /servers HTTP 404: + This particular logging code was added in the Zed release: + + https://review.opendev.org/c/openstack/nova/+/806683 + + Example log for GET /servers HTTP 404: Mar 28 01:11:57 ubuntu-focal [email protected][3890335]: INFO nova.api.openstack.wsgi [None req-f5dbcb16-318b-4b19-96a6-62492ac94677 demo demo] HTTP exception thrown: Instance test could not be found. Mar 28 01:11:57 ubuntu-focal [email protected][3890335]: DEBUG nova.api.openstack.wsgi [None req-f5dbcb16-318b-4b19-96a6-62492ac94677 demo demo] Request method failure captured: Mar 28 01:11:57 ubuntu-focal [email protected][3890335]: request: GET /compute/v2.1/servers/test HTTP/1.1 Mar 28 01:11:57 ubuntu-focal [email protected][3890335]: Accept: application/json Mar 28 01:11:57 ubuntu-focal [email protected][3890335]: Accept-Encoding: gzip, deflate Mar 28 01:11:57 ubuntu-focal [email protected][3890335]: Connection: keep-alive Mar 28 01:11:57 ubuntu-focal [email protected][3890335]: Content-Length: 0 Mar 28 01:11:57 ubuntu-focal [email protected][3890335]: Host: 192.168.44.11 Mar 28 01:11:57 ubuntu-focal [email protected][3890335]: Openstack-System-Scope: None Mar 28 01:11:57 ubuntu-focal [email protected][3890335]: User-Agent: python-novaclient Mar 28 01:11:57 ubuntu-focal [email protected][3890335]: X-Auth-Token: gAAAAABkIj7cGDJYKbnQalHot3qfNuAY1AMwMdKFP0kVQB6_8HRCSizDQpxMfspjx2S7t8rMWPSwYwg0-Yox2QM9E3KPWVq72YPl-cr8XwlDIn-ev9WjpmkmCtlOqOs0M0rOSvQggxdNB0xLx 2-gYiWUyMAYW6A1vXcup7Rvs8-YFetPKr2vGnY [...] Full log trace: https://paste.openstack.org/show/bx9CmbgOsDNrIn16PfUW ** Description changed: Noticed this while working on something else, if the API is going to return a non 2xx HTTP success status code, a lot of request details are logged including the user's unsanitized auth token. In the past, operators considered this to be a security issue despite logging only at level DEBUG. For this reason I am opening a bug for review. This particular logging code was added in the Zed release: https://review.opendev.org/c/openstack/nova/+/806683 + + These are logged a lot when using OSC + server names because OSC always + tries to lookup a name as a UUID (which will fail with 404) before it + falls back on trying it as an ID. So commands such as 'openstack server + show MyVM' will produce debug logs like the following. Example log for GET /servers HTTP 404: Mar 28 01:11:57 ubuntu-focal [email protected][3890335]: INFO nova.api.openstack.wsgi [None req-f5dbcb16-318b-4b19-96a6-62492ac94677 demo demo] HTTP exception thrown: Instance test could not be found. Mar 28 01:11:57 ubuntu-focal [email protected][3890335]: DEBUG nova.api.openstack.wsgi [None req-f5dbcb16-318b-4b19-96a6-62492ac94677 demo demo] Request method failure captured: Mar 28 01:11:57 ubuntu-focal [email protected][3890335]: request: GET /compute/v2.1/servers/test HTTP/1.1 Mar 28 01:11:57 ubuntu-focal [email protected][3890335]: Accept: application/json Mar 28 01:11:57 ubuntu-focal [email protected][3890335]: Accept-Encoding: gzip, deflate Mar 28 01:11:57 ubuntu-focal [email protected][3890335]: Connection: keep-alive Mar 28 01:11:57 ubuntu-focal [email protected][3890335]: Content-Length: 0 Mar 28 01:11:57 ubuntu-focal [email protected][3890335]: Host: 192.168.44.11 Mar 28 01:11:57 ubuntu-focal [email protected][3890335]: Openstack-System-Scope: None Mar 28 01:11:57 ubuntu-focal [email protected][3890335]: User-Agent: python-novaclient Mar 28 01:11:57 ubuntu-focal [email protected][3890335]: X-Auth-Token: gAAAAABkIj7cGDJYKbnQalHot3qfNuAY1AMwMdKFP0kVQB6_8HRCSizDQpxMfspjx2S7t8rMWPSwYwg0-Yox2QM9E3KPWVq72YPl-cr8XwlDIn-ev9WjpmkmCtlOqOs0M0rOSvQggxdNB0xLx 2-gYiWUyMAYW6A1vXcup7Rvs8-YFetPKr2vGnY [...] Full log trace: https://paste.openstack.org/show/bx9CmbgOsDNrIn16PfUW ** Description changed: Noticed this while working on something else, if the API is going to return a non 2xx HTTP success status code, a lot of request details are logged including the user's unsanitized auth token. In the past, operators considered this to be a security issue despite logging only at level DEBUG. For this reason I am opening a bug for review. This particular logging code was added in the Zed release: https://review.opendev.org/c/openstack/nova/+/806683 These are logged a lot when using OSC + server names because OSC always tries to lookup a name as a UUID (which will fail with 404) before it falls back on trying it as an ID. So commands such as 'openstack server - show MyVM' will produce debug logs like the following. + show MyServer' will produce debug logs like the following. Example log for GET /servers HTTP 404: Mar 28 01:11:57 ubuntu-focal [email protected][3890335]: INFO nova.api.openstack.wsgi [None req-f5dbcb16-318b-4b19-96a6-62492ac94677 demo demo] HTTP exception thrown: Instance test could not be found. Mar 28 01:11:57 ubuntu-focal [email protected][3890335]: DEBUG nova.api.openstack.wsgi [None req-f5dbcb16-318b-4b19-96a6-62492ac94677 demo demo] Request method failure captured: Mar 28 01:11:57 ubuntu-focal [email protected][3890335]: request: GET /compute/v2.1/servers/test HTTP/1.1 Mar 28 01:11:57 ubuntu-focal [email protected][3890335]: Accept: application/json Mar 28 01:11:57 ubuntu-focal [email protected][3890335]: Accept-Encoding: gzip, deflate Mar 28 01:11:57 ubuntu-focal [email protected][3890335]: Connection: keep-alive Mar 28 01:11:57 ubuntu-focal [email protected][3890335]: Content-Length: 0 Mar 28 01:11:57 ubuntu-focal [email protected][3890335]: Host: 192.168.44.11 Mar 28 01:11:57 ubuntu-focal [email protected][3890335]: Openstack-System-Scope: None Mar 28 01:11:57 ubuntu-focal [email protected][3890335]: User-Agent: python-novaclient Mar 28 01:11:57 ubuntu-focal [email protected][3890335]: X-Auth-Token: gAAAAABkIj7cGDJYKbnQalHot3qfNuAY1AMwMdKFP0kVQB6_8HRCSizDQpxMfspjx2S7t8rMWPSwYwg0-Yox2QM9E3KPWVq72YPl-cr8XwlDIn-ev9WjpmkmCtlOqOs0M0rOSvQggxdNB0xLx 2-gYiWUyMAYW6A1vXcup7Rvs8-YFetPKr2vGnY [...] Full log trace: https://paste.openstack.org/show/bx9CmbgOsDNrIn16PfUW -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/2012993 Title: Unredacted X-Auth-Token logged at level DEBUG in nova-api when HTTP status code != 2xx Status in OpenStack Compute (nova): New Bug description: Noticed this while working on something else, if the API is going to return a non 2xx HTTP success status code, a lot of request details are logged including the user's unsanitized auth token. In the past, operators considered this to be a security issue despite logging only at level DEBUG. For this reason I am opening a bug for review. This particular logging code was added in the Zed release: https://review.opendev.org/c/openstack/nova/+/806683 These are logged a lot when using OSC + server names because OSC always tries to lookup a name as a UUID (which will fail with 404) before it falls back on trying it as an ID. So commands such as 'openstack server show MyServer' will produce debug logs like the following. Example log for GET /servers HTTP 404: Mar 28 01:11:57 ubuntu-focal [email protected][3890335]: INFO nova.api.openstack.wsgi [None req-f5dbcb16-318b-4b19-96a6-62492ac94677 demo demo] HTTP exception thrown: Instance test could not be found. Mar 28 01:11:57 ubuntu-focal [email protected][3890335]: DEBUG nova.api.openstack.wsgi [None req-f5dbcb16-318b-4b19-96a6-62492ac94677 demo demo] Request method failure captured: Mar 28 01:11:57 ubuntu-focal [email protected][3890335]: request: GET /compute/v2.1/servers/test HTTP/1.1 Mar 28 01:11:57 ubuntu-focal [email protected][3890335]: Accept: application/json Mar 28 01:11:57 ubuntu-focal [email protected][3890335]: Accept-Encoding: gzip, deflate Mar 28 01:11:57 ubuntu-focal [email protected][3890335]: Connection: keep-alive Mar 28 01:11:57 ubuntu-focal [email protected][3890335]: Content-Length: 0 Mar 28 01:11:57 ubuntu-focal [email protected][3890335]: Host: 192.168.44.11 Mar 28 01:11:57 ubuntu-focal [email protected][3890335]: Openstack-System-Scope: None Mar 28 01:11:57 ubuntu-focal [email protected][3890335]: User-Agent: python-novaclient Mar 28 01:11:57 ubuntu-focal [email protected][3890335]: X-Auth-Token: gAAAAABkIj7cGDJYKbnQalHot3qfNuAY1AMwMdKFP0kVQB6_8HRCSizDQpxMfspjx2S7t8rMWPSwYwg0-Yox2QM9E3KPWVq72YPl-cr8XwlDIn-ev9WjpmkmCtlOqOs0M0rOSvQggxdNB0xLx 2-gYiWUyMAYW6A1vXcup7Rvs8-YFetPKr2vGnY [...] Full log trace: https://paste.openstack.org/show/bx9CmbgOsDNrIn16PfUW To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/2012993/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : [email protected] Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
