Re: [openstack-dev] [nova] [glance] [cinder] [neutron] [keystone] [heat] [telemetry] - RFC cross project request id tracking
On 05/15/2017 12:16 PM, Doug Hellmann wrote: > Excerpts from Zane Bitter's message of 2017-05-15 11:43:07 -0400: >> On 15/05/17 10:35, Doug Hellmann wrote: >>> Excerpts from Sean Dague's message of 2017-05-15 10:01:20 -0400: On 05/15/2017 09:35 AM, Doug Hellmann wrote: > Excerpts from Sean Dague's message of 2017-05-14 07:04:03 -0400: >> One of the things that came up in a logging Forum session is how much >> effort operators are having to put into reconstructing flows for things >> like server boot when they go wrong, as every time we jump a service >> barrier the request-id is reset to something new. The back and forth >> between Nova / Neutron and Nova / Glance would be definitely well served >> by this. Especially if this is something that's easy to query in elastic >> search. >> >> The last time this came up, some people were concerned that trusting >> request-id on the wire was concerning to them because it's coming from >> random users. We're going to assume that's still a concern by some. >> However, since the last time that came up, we've introduced the concept >> of "service users", which are a set of higher priv services that we are >> using to wrap user requests between services so that long running >> request chains (like image snapshot). We trust these service users >> enough to keep on trucking even after the user token has expired for >> this long run operations. We could use this same trust path for >> request-id chaining. >> >> So, the basic idea is, services will optionally take an inbound >> X-OpenStack-Request-ID which will be strongly validated to the format >> (req-$uuid). They will continue to always generate one as well. When the > > Do all of our services use that format for request ID? I thought Heat > used something added on to a UUID, or at least longer than a UUID? >> >> FWIW I don't recall ever hearing this. >> >> - ZB > > OK, maybe I'm mixing it up with some other field that we expected to be > a UUID and wasn't. Ignore me and proceed. :-) Given that the validation will be in a single function in oslo.middeware.request_id, if projects have other needs in the future, there will be a single knob to turn. However, starting strict to be req-$UUID eliminates a whole class of potential bugs and injection concerns. -Sean -- Sean Dague http://dague.net __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] [glance] [cinder] [neutron] [keystone] [heat] [telemetry] - RFC cross project request id tracking
Excerpts from Zane Bitter's message of 2017-05-15 11:43:07 -0400: > On 15/05/17 10:35, Doug Hellmann wrote: > > Excerpts from Sean Dague's message of 2017-05-15 10:01:20 -0400: > >> On 05/15/2017 09:35 AM, Doug Hellmann wrote: > >>> Excerpts from Sean Dague's message of 2017-05-14 07:04:03 -0400: > One of the things that came up in a logging Forum session is how much > effort operators are having to put into reconstructing flows for things > like server boot when they go wrong, as every time we jump a service > barrier the request-id is reset to something new. The back and forth > between Nova / Neutron and Nova / Glance would be definitely well served > by this. Especially if this is something that's easy to query in elastic > search. > > The last time this came up, some people were concerned that trusting > request-id on the wire was concerning to them because it's coming from > random users. We're going to assume that's still a concern by some. > However, since the last time that came up, we've introduced the concept > of "service users", which are a set of higher priv services that we are > using to wrap user requests between services so that long running > request chains (like image snapshot). We trust these service users > enough to keep on trucking even after the user token has expired for > this long run operations. We could use this same trust path for > request-id chaining. > > So, the basic idea is, services will optionally take an inbound > X-OpenStack-Request-ID which will be strongly validated to the format > (req-$uuid). They will continue to always generate one as well. When the > >>> > >>> Do all of our services use that format for request ID? I thought Heat > >>> used something added on to a UUID, or at least longer than a UUID? > > FWIW I don't recall ever hearing this. > > - ZB OK, maybe I'm mixing it up with some other field that we expected to be a UUID and wasn't. Ignore me and proceed. :-) Doug > > >> Don't know, now is a good time to speak up. > >> http://logs.openstack.org/85/464585/1/check/gate-heat-dsvm-functional-orig-mysql-lbaasv2-ubuntu-xenial/e1bca9e/logs/screen-h-eng.txt.gz#_2017-05-15_10_08_10_617 > >> seems to indicate that it's using the format everyone else is using. > >> > >> Swift does things a bit differently with suffixes, but they aren't using > >> the common middleware. > >> > >> I've done code look throughs on nova/glance/cinder/neutron/keystone, but > >> beyond that folks will need to speak up as to where this might break > >> down. At worst failing validation just means you end up in the old > >> (current) behavior. > >> > >> -Sean > >> > > > > OK. I vaguely remembered something from the early days of ceilometer > > trying to collect those notifications, but maybe I'm confusing it with > > something else. I've added [heat] to the subject line to get that team's > > attention for input. > > > > Doug > > > > __ > > OpenStack Development Mailing List (not for usage questions) > > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] [glance] [cinder] [neutron] [keystone] [heat] [telemetry] - RFC cross project request id tracking
On 15/05/17 10:35, Doug Hellmann wrote: Excerpts from Sean Dague's message of 2017-05-15 10:01:20 -0400: On 05/15/2017 09:35 AM, Doug Hellmann wrote: Excerpts from Sean Dague's message of 2017-05-14 07:04:03 -0400: One of the things that came up in a logging Forum session is how much effort operators are having to put into reconstructing flows for things like server boot when they go wrong, as every time we jump a service barrier the request-id is reset to something new. The back and forth between Nova / Neutron and Nova / Glance would be definitely well served by this. Especially if this is something that's easy to query in elastic search. The last time this came up, some people were concerned that trusting request-id on the wire was concerning to them because it's coming from random users. We're going to assume that's still a concern by some. However, since the last time that came up, we've introduced the concept of "service users", which are a set of higher priv services that we are using to wrap user requests between services so that long running request chains (like image snapshot). We trust these service users enough to keep on trucking even after the user token has expired for this long run operations. We could use this same trust path for request-id chaining. So, the basic idea is, services will optionally take an inbound X-OpenStack-Request-ID which will be strongly validated to the format (req-$uuid). They will continue to always generate one as well. When the Do all of our services use that format for request ID? I thought Heat used something added on to a UUID, or at least longer than a UUID? FWIW I don't recall ever hearing this. - ZB Don't know, now is a good time to speak up. http://logs.openstack.org/85/464585/1/check/gate-heat-dsvm-functional-orig-mysql-lbaasv2-ubuntu-xenial/e1bca9e/logs/screen-h-eng.txt.gz#_2017-05-15_10_08_10_617 seems to indicate that it's using the format everyone else is using. Swift does things a bit differently with suffixes, but they aren't using the common middleware. I've done code look throughs on nova/glance/cinder/neutron/keystone, but beyond that folks will need to speak up as to where this might break down. At worst failing validation just means you end up in the old (current) behavior. -Sean OK. I vaguely remembered something from the early days of ceilometer trying to collect those notifications, but maybe I'm confusing it with something else. I've added [heat] to the subject line to get that team's attention for input. Doug __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] [glance] [cinder] [neutron] [keystone] [heat] [telemetry] - RFC cross project request id tracking
Excerpts from Sean Dague's message of 2017-05-15 10:01:20 -0400: > On 05/15/2017 09:35 AM, Doug Hellmann wrote: > > Excerpts from Sean Dague's message of 2017-05-14 07:04:03 -0400: > >> One of the things that came up in a logging Forum session is how much > >> effort operators are having to put into reconstructing flows for things > >> like server boot when they go wrong, as every time we jump a service > >> barrier the request-id is reset to something new. The back and forth > >> between Nova / Neutron and Nova / Glance would be definitely well served > >> by this. Especially if this is something that's easy to query in elastic > >> search. > >> > >> The last time this came up, some people were concerned that trusting > >> request-id on the wire was concerning to them because it's coming from > >> random users. We're going to assume that's still a concern by some. > >> However, since the last time that came up, we've introduced the concept > >> of "service users", which are a set of higher priv services that we are > >> using to wrap user requests between services so that long running > >> request chains (like image snapshot). We trust these service users > >> enough to keep on trucking even after the user token has expired for > >> this long run operations. We could use this same trust path for > >> request-id chaining. > >> > >> So, the basic idea is, services will optionally take an inbound > >> X-OpenStack-Request-ID which will be strongly validated to the format > >> (req-$uuid). They will continue to always generate one as well. When the > > > > Do all of our services use that format for request ID? I thought Heat > > used something added on to a UUID, or at least longer than a UUID? > > Don't know, now is a good time to speak up. > http://logs.openstack.org/85/464585/1/check/gate-heat-dsvm-functional-orig-mysql-lbaasv2-ubuntu-xenial/e1bca9e/logs/screen-h-eng.txt.gz#_2017-05-15_10_08_10_617 > seems to indicate that it's using the format everyone else is using. > > Swift does things a bit differently with suffixes, but they aren't using > the common middleware. > > I've done code look throughs on nova/glance/cinder/neutron/keystone, but > beyond that folks will need to speak up as to where this might break > down. At worst failing validation just means you end up in the old > (current) behavior. > > -Sean > OK. I vaguely remembered something from the early days of ceilometer trying to collect those notifications, but maybe I'm confusing it with something else. I've added [heat] to the subject line to get that team's attention for input. Doug __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
