Excerpts from Kekane, Abhishek's message of 2015-05-27 07:06:56 +0000: > Hi Devs, > > Each OpenStack service sends a request ID header with HTTP responses. This > request ID can be useful for tracking down problems in the logs. However, > when operation crosses service boundaries, this tracking can become > difficult, as each service has its own request ID. Request ID is not returned > to the caller, so it is not easy to track the request. This becomes > especially problematic when requests are coming in parallel. For example, > glance will call cinder for creating image, but that cinder instance may be > handling several other requests at the same time. By using same request ID in > the log, user can easily find the cinder request ID that is same as glance > request ID in the g-api log. It will help operators/developers to analyse > logs effectively. > > To address this issue we have come up with following solutions: > > Solution 1: Return tuple containing headers and body from respective clients > (also favoured by Joe Gordon) > Reference: > https://review.openstack.org/#/c/156508/6/specs/log-request-id-mappings.rst > > Pros: > 1. Maintains backward compatibility > 2. Effective debugging/analysing of the problem as both calling service > request-id and called service request-id are logged in same log message > 3. Build a full call graph > 4. End user will able to know the request-id of the request and can approach > service provider to know the cause of failure of particular request. > > Cons: > 1. The changes need to be done first in cross-projects before making changes > in clients > 2. Applications which are using python-*clients needs to do required changes > (check return type of response) > > > Solution 2: Use thread local storage to store 'x-openstack-request-id' > returned from headers (suggested by Doug Hellmann) > Reference: > https://review.openstack.org/#/c/156508/9/specs/log-request-id-mappings.rst > > Add new method 'get_openstack_request_id' to return this request-id to the > caller. > > Pros: > 1. Doesn't break compatibility > 2. Minimal changes are required in client > 3. Build a full call graph > > Cons: > 1. Malicious user can send long request-id to fill up the disk-space, > resulting in potential DoS > 2. Changes need to be done in all python-*clients > 3. Last request id should be flushed out in a subsequent call otherwise it > will return wrong request id to the caller > > > Solution 3: Unique request-id across OpenStack Services (suggested by Jamie > Lennox) > Reference: > https://review.openstack.org/#/c/156508/10/specs/log-request-id-mappings.rst > > Get 'x-openstack-request-id' from auth plugin and add it to the request > headers. If 'x-openstack-request-id' key is present in the request header, > then it will use the same one further or else it will generate a new one. > > Dependencies: > https://review.openstack.org/#/c/164582/ - Include request-id in auth plugin > and add it to request headers > https://review.openstack.org/#/c/166063/ - Add session-object for glance > client > Add 'UserAuthPlugin' and '_ContextAuthPlugin' same as nova in cinder and > neutron > > > Pros: > 1. Using same request id for the request crossing multiple service boundaries > will help operators/developers identify the problem quickly > 2. Required changes only in keystonemiddleware and oslo_middleware libraries. > No changes are required in the python client bindings or OpenStack core > services > > Cons: > 1. As 'x-openstack-request-id' in the request header will be visible to the > user, it is possible to send same request id for multiple requests which in > turn could create more problems in case of troubleshooting cause of the > failure as request_id middleware will not check for its uniqueness in the > scope of the running OpenStack service. > 2. Having the same request ID for all services for a single user API call > means you cannot generate a full call graph. For example if a single user's > nova API call produces 2 calls to glance you want to be able to differentiate > the two different calls. > > > During the Liberty design summit, I had a chance of discussing these designs > with some of the core members like Doug, Joe Gordon, Jamie Lennox etc. But > not able to came to any conclusion on the final design and know the > communities direction by which way they want to use this request-id > effectively. > > However IMO, solution 1 sounds more useful as the debugger can able to build > the full call graph which can be helpful for analysing gate failures > effectively as well as end user will be able to know his request-id and can > track his request. > > I request all community members to go through these solutions and let us know > which is the appropriate way to improve the logs by logging request-id.
Robert Collins pointed out that os-profiler is already tracking requests across REST calls. Does it use one of the proposed methods? Since os-profiler is seeing more adoption, could we combine efforts here? Doug __________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: [email protected]?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
