(2013/12/06 17:57), Joe Gordon wrote:
> On Dec 6, 2013 9:57 AM, "Maru Newby" <ma...@redhat.com 
> <mailto:ma...@redhat.com>> wrote:
>  >
>  >
>  > On Dec 6, 2013, at 1:09 AM, John Dickinson <m...@not.mn 
> <mailto:m...@not.mn>> wrote:
>  >
>  > >
>  > > On Dec 5, 2013, at 1:36 AM, Maru Newby <ma...@redhat.com 
> <mailto:ma...@redhat.com>> wrote:
>  > >
>  > >>
>  > >> On Dec 3, 2013, at 12:18 AM, Joe Gordon <joe.gord...@gmail.com 
> <mailto:joe.gord...@gmail.com>> wrote:
>  > >>
>  > >>>
>  > >>>
>  > >>>
>  > >>> On Sun, Dec 1, 2013 at 7:04 PM, John Dickinson <m...@not.mn 
> <mailto:m...@not.mn>> wrote:
>  > >>> Just to add to the story, Swift uses "X-Trans-Id" and generates it in 
> the outer-most "catch_errors" middleware.
>  > >>>
>  > >>> Swift's catch errors middleware is responsible for ensuring that the 
> transaction id exists on each request, and that all errors previously 
> uncaught, anywhere in the pipeline, are caught and logged. If there is not a 
> common way to do this, yet, I submit it as a great template for solving this
> problem. It's simple, scalable, and well-tested (ie tests and running in prod 
> for years).
>  > >>>
>  > >>> 
> https://github.com/openstack/swift/blob/master/swift/common/middleware/catch_errors.py
>  > >>>
>  > >>> Leaving aside error handling and only focusing on the transaction id 
> (or request id) generation, since OpenStack services are exposed to untrusted 
> clients, how would you propose communicating the appropriate transaction id 
> to a different service? I can see great benefit to having a glance
> transaction ID carry through to Swift requests (and so on), but how should 
> the transaction id be communicated? It's not sensitive info, but I can 
> imagine a pretty big problem when trying to track down errors if a client 
> application decides to set eg the X-Set-Transaction-Id header on every request
> to the same thing.
>  > >>>
>  > >>> -1 to cross service request IDs, for the reasons John mentions above.
>  > >>>
>  > >>>
>  > >>> Thanks for bringing this up, and I'd welcome a patch in Swift that 
> would use a common library to generate the transaction id, if it were 
> installed. I can see that there would be huge advantage to operators to trace 
> requests through multiple systems.
>  > >>>
>  > >>> Another option would be for each system that calls an another 
> OpenStack system to expect and log the transaction ID for the request that 
> was given. This would be looser coupling and be more forgiving for a 
> heterogeneous cluster. Eg when Glance makes a call to Swift, Glance cloud log 
> the
> transaction id that Swift used (from the Swift response). Likewise, when 
> Swift makes a call to Keystone, Swift could log the Keystone transaction id. 
> This wouldn't result in a single transaction id across all systems, but it 
> would provide markers so an admin could trace the request.
>  > >>>
>  > >>> There was a session on this at the summit, and although the notes are 
> a little scarce this was the conclusion we came up with.  Every time a cross 
> service call is made, we will log and send a notification for ceilometer to 
> consume, with the request-ids of both request ids.  One of the
> benefits of this approach is that we can easily generate a tree of all the 
> API calls that are made (and clearly show when multiple calls are made to the 
> same service), something that just a cross service request id would have 
> trouble with.
>  > >>
>  > >> Is wise to trust anything a client provides to ensure traceability?  If 
> a user receives a request id back from Nova, then submits that request id in 
> an unrelated request to Neutron, the traceability would be effectively 
> corrupted.  If the consensus is that we don't want to securely deliver
> request ids for inter-service calls, how about requiring a service to log its 
> request id along with the request id returned from a call to another service 
> to achieve the a similar result?
>  > >
>  > > Yes, this is what I was proposing. I think this is the best path forward.

I think Logging returned request-id at client side is the best way.
We can track each API call even when multiple API calls are invoked in one API 
 > Nova does this for glance client today.  Glance client logs the out glances 
 > request Id and that message is wrapped with novas request id.

When I ran devstack, glance client log (with glance request-id) is not wrapped
with nova request-id: http://paste.openstack.org/show/54590/
It is because glanceclient uses standard logging instead of 
It is common to all client libraries and I am not sure why client libraries
do not use openstack.common.log.
> Ceilometer wanted notifications of these events as well so it could track 
> things better.

Should this feature a part of client libraries or server side which calls 
client libraries?
At now client libraries do not return information about response headers 
including request-id.
If we support it in server side, a standard way to return request-id in a 
response to a caller.
I don't have good idea on it at the moment and it is just a question.


>  >
>  > Ok, great.  And as per your suggestion, a middleware-based error handler 
> will soon be proposed for oslo that will secondarily ensure that a request id 
> is added to the request.
>  >
>  > >
>  > >
>  > >> The catch is that every call point (or client instantiation?) would 
> have to be modified to pass the request id instead of just logging at one 
> place in each service.  Is that a cost worth paying?
>  > >
>  > > Perhaps this is my ignorance of how other projects work today, but does 
> this not already happen? Is it possible to get a response from an API call to 
> an OpenStack project that doesn't include a request id?
>  >
>  > We'll have it in Neutron real-soon-now, and then I think the answer will 
> be 'yes'.
>  >
>  > On reflection, it should be easy enough for a given service to ensure that 
> calls to other services are automatically instrumented to log request id 
> pairs.  Again, probably something for oslo.
>  >
>  >
>  > m.
>  > _______________________________________________
>  > OpenStack-dev mailing list
>  > OpenStack-dev@lists.openstack.org 
> <mailto:OpenStack-dev@lists.openstack.org>
>  > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
OpenStack-dev mailing list

Reply via email to