(2013/12/06 17:57), Joe Gordon wrote: > > On Dec 6, 2013 9:57 AM, "Maru Newby" <ma...@redhat.com > <mailto:ma...@redhat.com>> wrote: > > > > > > On Dec 6, 2013, at 1:09 AM, John Dickinson <m...@not.mn > <mailto:m...@not.mn>> wrote: > > > > > > > > On Dec 5, 2013, at 1:36 AM, Maru Newby <ma...@redhat.com > <mailto:ma...@redhat.com>> wrote: > > > > > >> > > >> On Dec 3, 2013, at 12:18 AM, Joe Gordon <joe.gord...@gmail.com > <mailto:joe.gord...@gmail.com>> wrote: > > >> > > >>> > > >>> > > >>> > > >>> On Sun, Dec 1, 2013 at 7:04 PM, John Dickinson <m...@not.mn > <mailto:m...@not.mn>> wrote: > > >>> Just to add to the story, Swift uses "X-Trans-Id" and generates it in > the outer-most "catch_errors" middleware. > > >>> > > >>> Swift's catch errors middleware is responsible for ensuring that the > transaction id exists on each request, and that all errors previously > uncaught, anywhere in the pipeline, are caught and logged. If there is not a > common way to do this, yet, I submit it as a great template for solving this > problem. It's simple, scalable, and well-tested (ie tests and running in prod > for years). > > >>> > > >>> > https://github.com/openstack/swift/blob/master/swift/common/middleware/catch_errors.py > > >>> > > >>> Leaving aside error handling and only focusing on the transaction id > (or request id) generation, since OpenStack services are exposed to untrusted > clients, how would you propose communicating the appropriate transaction id > to a different service? I can see great benefit to having a glance > transaction ID carry through to Swift requests (and so on), but how should > the transaction id be communicated? It's not sensitive info, but I can > imagine a pretty big problem when trying to track down errors if a client > application decides to set eg the X-Set-Transaction-Id header on every request > to the same thing. > > >>> > > >>> -1 to cross service request IDs, for the reasons John mentions above. > > >>> > > >>> > > >>> Thanks for bringing this up, and I'd welcome a patch in Swift that > would use a common library to generate the transaction id, if it were > installed. I can see that there would be huge advantage to operators to trace > requests through multiple systems. > > >>> > > >>> Another option would be for each system that calls an another > OpenStack system to expect and log the transaction ID for the request that > was given. This would be looser coupling and be more forgiving for a > heterogeneous cluster. Eg when Glance makes a call to Swift, Glance cloud log > the > transaction id that Swift used (from the Swift response). Likewise, when > Swift makes a call to Keystone, Swift could log the Keystone transaction id. > This wouldn't result in a single transaction id across all systems, but it > would provide markers so an admin could trace the request. > > >>> > > >>> There was a session on this at the summit, and although the notes are > a little scarce this was the conclusion we came up with. Every time a cross > service call is made, we will log and send a notification for ceilometer to > consume, with the request-ids of both request ids. One of the > benefits of this approach is that we can easily generate a tree of all the > API calls that are made (and clearly show when multiple calls are made to the > same service), something that just a cross service request id would have > trouble with. > > >> > > >> Is wise to trust anything a client provides to ensure traceability? If > a user receives a request id back from Nova, then submits that request id in > an unrelated request to Neutron, the traceability would be effectively > corrupted. If the consensus is that we don't want to securely deliver > request ids for inter-service calls, how about requiring a service to log its > request id along with the request id returned from a call to another service > to achieve the a similar result? > > > > > > Yes, this is what I was proposing. I think this is the best path forward.
I think Logging returned request-id at client side is the best way. We can track each API call even when multiple API calls are invoked in one API request. > > Nova does this for glance client today. Glance client logs the out glances > request Id and that message is wrapped with novas request id. When I ran devstack, glance client log (with glance request-id) is not wrapped with nova request-id: http://paste.openstack.org/show/54590/ It is because glanceclient uses standard logging instead of openstack.common.log. It is common to all client libraries and I am not sure why client libraries do not use openstack.common.log. > > Ceilometer wanted notifications of these events as well so it could track > things better. Should this feature a part of client libraries or server side which calls client libraries? At now client libraries do not return information about response headers including request-id. If we support it in server side, a standard way to return request-id in a response to a caller. I don't have good idea on it at the moment and it is just a question. Thanks, Akihiro > > > > > Ok, great. And as per your suggestion, a middleware-based error handler > will soon be proposed for oslo that will secondarily ensure that a request id > is added to the request. > > > > > > > > > > >> The catch is that every call point (or client instantiation?) would > have to be modified to pass the request id instead of just logging at one > place in each service. Is that a cost worth paying? > > > > > > Perhaps this is my ignorance of how other projects work today, but does > this not already happen? Is it possible to get a response from an API call to > an OpenStack project that doesn't include a request id? > > > > We'll have it in Neutron real-soon-now, and then I think the answer will > be 'yes'. > > > > On reflection, it should be easy enough for a given service to ensure that > calls to other services are automatically instrumented to log request id > pairs. Again, probably something for oslo. > > > > > > m. > > _______________________________________________ > > OpenStack-dev mailing list > > OpenStack-dev@lists.openstack.org > <mailto:OpenStack-dev@lists.openstack.org> > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > _______________________________________________ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
