Hi, According to the thread of request-id [1], there seems a need for a wsgi middleware which generates request-id for each REST request and ensures it in a corresponding response. To do this, the middlware is located outer-most of a wsgi pipeline and needs to catch all kind of exceptions. It is the concept swift already has as catch_error middleware.
How about adding such middlware to oslo? Ensuring request-id in log and API response is a common requirement across OpenStack projects but each project implements in different ways now. I was suggested from Maru on IRC too. The feature of the proposed middleware is simple: * Catch all exceptions in wsgi pipeline * Generate request-id for each REST request * Set the generated request-id to request environment * Add X-OpenStack-Request-Id header to API response to return request-id * Expected to place outer-most (i.e., first) in wsgi pipeline The initial implemetation proposed to neutron is available at https://review.openstack.org/#/c/58270/ (The name of middleware is not good....) Regarding implementation, a question is whether webob can be used or not. Now request-id is generated in Context class and Context is instantiated in auth middlware in most projects like nova, cinder, neutron. We need to change auth middlware so that request-id will be retreived from request env. Context class and auth middlware are also similar but different acorss projects. It may time to update/add these classes in oslo. Regards, Akihiro [1] http://lists.openstack.org/pipermail/openstack-dev/2013-November/thread.html#20683 (2013/12/02 23:47), Jay Pipes wrote: > On 12/01/2013 10:04 PM, John Dickinson wrote: >> Just to add to the story, Swift uses "X-Trans-Id" and generates it in >> the outer-most "catch_errors" middleware. >> >> Swift's catch errors middleware is responsible for ensuring that the >> transaction id exists on each request, and that all errors previously >> uncaught, anywhere in the pipeline, are caught and logged. If there >> is not a common way to do this, yet, I submit it as a great template >> for solving this problem. It's simple, scalable, and well-tested (ie >> tests and running in prod for years). >> >> > https://github.com/openstack/swift/blob/master/swift/common/middleware/cat > ch_errors.py > > ++ > > If there's prior art here, might as well use it. I'm not a huge fan of > using the term "transaction" within things that do not have a > transactional safety context... but that's just because of my background > in RDBMS stuff. If X-Trans-Id is already in use by another OpenStack > project, it should probably take precedence over something new unless > there is a really good reason otherwise (and my personal opinion about > the semantics of transactions ain't a good reason!). > >> Leaving aside error handling and only focusing on the transaction id >> (or request id) generation, since OpenStack services are exposed to >> untrusted clients, how would you propose communicating the >> appropriate transaction id to a different service? I can see great >> benefit to having a glance transaction ID carry through to Swift >> requests (and so on), but how should the transaction id be >> communicated? It's not sensitive info, but I can imagine a pretty big >> problem when trying to track down errors if a client application >> decides to set eg the X-Set-Transaction-Id header on every request to >> the same thing. > > I suppose if this were really a problem (and I'm not sold on the idea > that it is a problem), one solution might be to store a checksum > somewhere for the transaction ID and some other piece of data. But I > don't really see that as super useful, and it would slow things down. > Glance already stores a checksum for important things like the data in > an image. If a service needs to be absolutely sure that a piece of data > hasn't been messed with, this cross-service request ID probably isn't > the thing to use... > >> Thanks for bringing this up, and I'd welcome a patch in Swift that >> would use a common library to generate the transaction id, if it were >> installed. I can see that there would be huge advantage to operators >> to trace requests through multiple systems. > > Hmm, so does that mean that you'd be open to (gradually) moving to an > x-openstack-request-id header to replace x-trans-id? > >> Another option would be for each system that calls an another >> OpenStack system to expect and log the transaction ID for the request >> that was given. This would be looser coupling and be more forgiving >> for a heterogeneous cluster. Eg when Glance makes a call to Swift, >> Glance cloud log the transaction id that Swift used (from the Swift >> response). Likewise, when Swift makes a call to Keystone, Swift could >> log the Keystone transaction id. This wouldn't result in a single >> transaction id across all systems, but it would provide markers so an >> admin could trace the request. > > Sure, this is a perfectly fine option, but doesn't really provide the > single traceable ID value that I think we're looking for here. > > Best, > -jay > >> On Dec 1, 2013, at 5:48 PM, Maru Newby <ma...@redhat.com> wrote: >> >>> >>> On Nov 30, 2013, at 1:00 AM, Sean Dague <s...@dague.net> wrote: >>> >>>> On 11/29/2013 10:33 AM, Jay Pipes wrote: >>>>> On 11/28/2013 07:45 AM, Akihiro Motoki wrote: >>>>>> Hi, >>>>>> >>>>>> I am working on adding request-id to API response in >>>>>> Neutron. After I checked what header is used in other >>>>>> projects header name varies project by project. It seems >>>>>> there is no consensus what header is recommended and it is >>>>>> better to have some consensus. >>>>>> >>>>>> nova: x-compute-request-id cinder: >>>>>> x-compute-request-id glance: x-openstack-request-id >>>>>> neutron: x-network-request-id (under review) >>>>>> >>>>>> request-id is assigned and used inside of each project now, >>>>>> so x-<service>-request-id looks good. On the other hand, if >>>>>> we have a plan to enhance request-id across projects, >>>>>> x-openstack-request-id looks better. >>>>> >>>>> My vote is for: >>>>> >>>>> x-openstack-request-id >>>>> >>>>> With an implementation of "create a request UUID if none exists >>>>> yet" in some standardized WSGI middleware... >>>> >>>> Agreed. I don't think I see any value in having these have >>>> different service names, having just x-openstack-request-id >>>> across all the services seems a far better idea, and come back >>>> through and fix nova and cinder to be that as well. >>> >>> +1 >>> >>> An openstack request id should be service agnostic to allow >>> tracking of a request across many services (e.g. a call to nova to >>> boot a VM should generate a request id that is provided to other >>> services in requests to provision said VM). All services would >>> ideally share a facility for generating new request ids and for >>> securely accepting request ids from other services. >>> >>> >>> m. >>> >>>> >>>> -Sean >>>> >>>> -- Sean Dague http://dague.net >>>> >>>> _______________________________________________ OpenStack-dev >>>> mailing list OpenStack-dev@lists.openstack.org >>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>> >>> >>> >>>> > _______________________________________________ >>> OpenStack-dev mailing list OpenStack-dev@lists.openstack.org >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> >> >> >> _______________________________________________ OpenStack-dev mailing >> list OpenStack-dev@lists.openstack.org >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> > > > _______________________________________________ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > _______________________________________________ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev