On Dec 6, 2013, at 1:09 AM, John Dickinson <m...@not.mn> wrote:

> On Dec 5, 2013, at 1:36 AM, Maru Newby <ma...@redhat.com> wrote:
>> On Dec 3, 2013, at 12:18 AM, Joe Gordon <joe.gord...@gmail.com> wrote:
>>> On Sun, Dec 1, 2013 at 7:04 PM, John Dickinson <m...@not.mn> wrote:
>>> Just to add to the story, Swift uses "X-Trans-Id" and generates it in the 
>>> outer-most "catch_errors" middleware.
>>> Swift's catch errors middleware is responsible for ensuring that the 
>>> transaction id exists on each request, and that all errors previously 
>>> uncaught, anywhere in the pipeline, are caught and logged. If there is not 
>>> a common way to do this, yet, I submit it as a great template for solving 
>>> this problem. It's simple, scalable, and well-tested (ie tests and running 
>>> in prod for years).
>>> https://github.com/openstack/swift/blob/master/swift/common/middleware/catch_errors.py
>>> Leaving aside error handling and only focusing on the transaction id (or 
>>> request id) generation, since OpenStack services are exposed to untrusted 
>>> clients, how would you propose communicating the appropriate transaction id 
>>> to a different service? I can see great benefit to having a glance 
>>> transaction ID carry through to Swift requests (and so on), but how should 
>>> the transaction id be communicated? It's not sensitive info, but I can 
>>> imagine a pretty big problem when trying to track down errors if a client 
>>> application decides to set eg the X-Set-Transaction-Id header on every 
>>> request to the same thing.
>>> -1 to cross service request IDs, for the reasons John mentions above.
>>> Thanks for bringing this up, and I'd welcome a patch in Swift that would 
>>> use a common library to generate the transaction id, if it were installed. 
>>> I can see that there would be huge advantage to operators to trace requests 
>>> through multiple systems.
>>> Another option would be for each system that calls an another OpenStack 
>>> system to expect and log the transaction ID for the request that was given. 
>>> This would be looser coupling and be more forgiving for a heterogeneous 
>>> cluster. Eg when Glance makes a call to Swift, Glance cloud log the 
>>> transaction id that Swift used (from the Swift response). Likewise, when 
>>> Swift makes a call to Keystone, Swift could log the Keystone transaction 
>>> id. This wouldn't result in a single transaction id across all systems, but 
>>> it would provide markers so an admin could trace the request.
>>> There was a session on this at the summit, and although the notes are a 
>>> little scarce this was the conclusion we came up with.  Every time a cross 
>>> service call is made, we will log and send a notification for ceilometer to 
>>> consume, with the request-ids of both request ids.  One of the benefits of 
>>> this approach is that we can easily generate a tree of all the API calls 
>>> that are made (and clearly show when multiple calls are made to the same 
>>> service), something that just a cross service request id would have trouble 
>>> with.
>> Is wise to trust anything a client provides to ensure traceability?  If a 
>> user receives a request id back from Nova, then submits that request id in 
>> an unrelated request to Neutron, the traceability would be effectively 
>> corrupted.  If the consensus is that we don't want to securely deliver 
>> request ids for inter-service calls, how about requiring a service to log 
>> its request id along with the request id returned from a call to another 
>> service to achieve the a similar result?
> Yes, this is what I was proposing. I think this is the best path forward.

Ok, great.  And as per your suggestion, a middleware-based error handler will 
soon be proposed for oslo that will secondarily ensure that a request id is 
added to the request.  

>> The catch is that every call point (or client instantiation?) would have to 
>> be modified to pass the request id instead of just logging at one place in 
>> each service.  Is that a cost worth paying?
> Perhaps this is my ignorance of how other projects work today, but does this 
> not already happen? Is it possible to get a response from an API call to an 
> OpenStack project that doesn't include a request id?

We'll have it in Neutron real-soon-now, and then I think the answer will be 

On reflection, it should be easy enough for a given service to ensure that 
calls to other services are automatically instrumented to log request id pairs. 
 Again, probably something for oslo.

OpenStack-dev mailing list

Reply via email to