Re: [openstack-dev] [all] cross project communication: Return request-id to caller
On 09:50 Dec 22, Doug Hellmann wrote: > Excerpts from Kekane, Abhishek's message of 2015-12-18 06:27:59 +: > > My point here is, should we follow this new approach in all python-clients > > to maintain the consistency across openstack? > > The original design was discussed, reviewed, and if I understand > correctly is already mostly implemented. Let's not change that. +1 this will just create confusion and more work of what's already out there. I would encourage folks to join us in the cross-project meetings where these specs are discussed [1] before they are rolled out to projects. [1] - https://wiki.openstack.org/wiki/Meetings/CrossProjectMeeting -- Mike Perez __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] cross project communication: Return request-id to caller
Excerpts from Kekane, Abhishek's message of 2015-12-18 06:27:59 +:
> Hi Devs,
>
> I have submitted a cross-project specs [1] for returning request-id to the
> caller which was approved. For implementation we have submitted patches in
> python-cinderclient [2] as per design in cross-project specs. However we have
> found one issue while implementing this design in python-glanceclient and
> submitted a lite-spec [3] to address this issue.
>
> As per the comments [4] on lite-specs, glance core are suggesting new
> approach to use hooks and pass empty list to each of the api, so that when
> the hook is executed it will append the request-id to the list.
>
> POC code for approach suggested by glance:
> --
> # change in python-glanceclient/v2/images.py to delete api
>
> import functools
>
> def append_request_id(req_id_lst, response, *args, **kwargs):
> req_id = response.headers.get('x-openstack-request-id')
> if req_id:
> req_id_lst.append(req_id)
>
> def setup_request_id_hook(req_id_lst):
> if req_id_lst is not None:
> return dict(response=functools.partial(append_request_id, req_id_lst))
>
> def delete(self, image_id, request_ids=None):
> hook = setup_request_id_hook(request_ids)
> url = '/v2/images/%s' % image_id
> self.http_client.delete(url, hooks=hook)
>
> # modify the do_image_delete function (glanceclient/v2/shell.py) to look as
> follows:
>
> request_ids = []
> gc.images.delete(args_id, request_ids=request_ids)
> print request_ids
>
> $ glance image-delete 11887d68-7faf-4821-9a42-3ec63451067c
> ['req-a0892f56-2626-4069-8787-f8bf56e0c1cc']
>
> We have tested this approach and it is working as per expectation, only thing
> is that we need to make changes in every method to add hook and request_id
> list.
> From third-party tools, user need to pass this request-ids list as mentioned
> in above poc in order to get the request-id back.
>
> My point here is, should we follow this new approach in all python-clients to
> maintain the consistency across openstack?
The original design was discussed, reviewed, and if I understand
correctly is already mostly implemented. Let's not change that.
Doug
>
> Please provide your feedback for the same.
>
> [1]
> http://specs.openstack.org/openstack/openstack-specs/specs/return-request-id.html
> [2] https://review.openstack.org/#/c/257170/ (base patch, followed by
> dependent patches)
> [3] https://bugs.launchpad.net/glance/+bug/1525259
> [4] https://bugs.launchpad.net/glance/+bug/1525259/comments/1 and
> https://bugs.launchpad.net/glance/+bug/1525259/comments/5
>
>
> Thank you,
>
> Abhishek Kekane
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [all] cross project communication: Return request-id to caller
Hi Devs,
I have submitted a cross-project specs [1] for returning request-id to the
caller which was approved. For implementation we have submitted patches in
python-cinderclient [2] as per design in cross-project specs. However we have
found one issue while implementing this design in python-glanceclient and
submitted a lite-spec [3] to address this issue.
As per the comments [4] on lite-specs, glance core are suggesting new approach
to use hooks and pass empty list to each of the api, so that when the hook is
executed it will append the request-id to the list.
POC code for approach suggested by glance:
--
# change in python-glanceclient/v2/images.py to delete api
import functools
def append_request_id(req_id_lst, response, *args, **kwargs):
req_id = response.headers.get('x-openstack-request-id')
if req_id:
req_id_lst.append(req_id)
def setup_request_id_hook(req_id_lst):
if req_id_lst is not None:
return dict(response=functools.partial(append_request_id, req_id_lst))
def delete(self, image_id, request_ids=None):
hook = setup_request_id_hook(request_ids)
url = '/v2/images/%s' % image_id
self.http_client.delete(url, hooks=hook)
# modify the do_image_delete function (glanceclient/v2/shell.py) to look as
follows:
request_ids = []
gc.images.delete(args_id, request_ids=request_ids)
print request_ids
$ glance image-delete 11887d68-7faf-4821-9a42-3ec63451067c
['req-a0892f56-2626-4069-8787-f8bf56e0c1cc']
We have tested this approach and it is working as per expectation, only thing
is that we need to make changes in every method to add hook and request_id list.
>From third-party tools, user need to pass this request-ids list as mentioned
>in above poc in order to get the request-id back.
My point here is, should we follow this new approach in all python-clients to
maintain the consistency across openstack?
Please provide your feedback for the same.
[1]
http://specs.openstack.org/openstack/openstack-specs/specs/return-request-id.html
[2] https://review.openstack.org/#/c/257170/ (base patch, followed by
dependent patches)
[3] https://bugs.launchpad.net/glance/+bug/1525259
[4] https://bugs.launchpad.net/glance/+bug/1525259/comments/1 and
https://bugs.launchpad.net/glance/+bug/1525259/comments/5
Thank you,
Abhishek Kekane
__
Disclaimer: This email and any attachments are sent in strictest confidence
for the sole use of the addressee and may contain legally privileged,
confidential, and proprietary data. If you are not the intended recipient,
please advise the sender by replying promptly to this email and then delete
and destroy this email and any attachments without any further use, copying
or forwarding.__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] cross project communication: Return request-id to caller
Hi Devs, I have modified the cross-project specs for request-id based on this analysis and submitted same for review. Please refer, https://review.openstack.org/#/c/156508/17/specs/return-request-id.rst and give your valuable feedback on the same. Thank you in advance. Abhishek Kekane -Original Message- From: Gorka Eguileor [mailto:gegui...@redhat.com] Sent: 29 July 2015 15:07 To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [all] cross project communication: Return request-id to caller On Tue, Jul 28, 2015 at 09:48:25AM -0400, Doug Hellmann wrote: Excerpts from Gorka Eguileor's message of 2015-07-28 10:37:42 +0200: On Fri, Jul 24, 2015 at 10:08:45AM -0400, Doug Hellmann wrote: Excerpts from Kekane, Abhishek's message of 2015-07-24 06:33:00 +: Hi Devs, X-Openstack-Request-Id. We have analysed python-cinderclient, python-glanceclient, python-novaclient, python-keystoneclient and python-neutronclient to check the return types. There are 9 ways return values are returned from python clients: 1. List 2. Dict 3. Resource class object 4. None 5. Tuple 6. Exception 7. Boolean (True/False, for keystoneclient) 8. Generator (for list api's in glanceclient) 9. String (for novaclient) Out of 9 we have solution for all return types except generator. In case of glance-client list api's are returning generator which is immutable. So it is not possible to return request-id in this case, which is a blocker for adopting the solution. I have added detail analysis for above return types in etherpad [2] as solution #3. If you have any suggestion in case of generation type then please let me know. It should be possible to create a new class to wrap the existing generator and implement the iterator protocol [3]. [3] https://docs.python.org/2/reference/expressions.html#generator-ite rator-methods Doug Unless I'm missing something I think we wouldn't even need to create a new class that implements the iterator protocol, we can just return a generator that generates from the other one. For example, for each of the requests, if we get the generator in variable *result* that returns dictionaries and we want to add *headers* to each dictionary: return (DictWithHeaders(resource, headers) for resource in result) Wouldn't that work? That would work, but it wouldn't be consistent with the way I read [2] as describing how the other methods are being updated. For example, a method that now returns a list() will return a ListWithHeaders(), and only that returned object will have the headers, not its contents. A caller could do: You are right, it wouldn't be consistent with other methods, and that's not good. So the new iterator class wrapper seems to be the way to go. Gorka. response = client.some_method_returning_a_list() print reponse.headers but could not do print response[0].headers and get the same values. Creating a GeneratorWithHeaders class mirrors that behavior for methods that return generators instead of lists. Doug [2] https://etherpad.openstack.org/p/request-id __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ Disclaimer: This email and any attachments are sent in strictest confidence for the sole use of the addressee and may contain legally privileged, confidential, and proprietary data. If you are not the intended recipient, please advise the sender by replying promptly to this email and then delete and destroy this email and any attachments without any further use, copying or forwarding. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] cross project communication: Return request-id to caller
On Tue, Jul 28, 2015 at 09:48:25AM -0400, Doug Hellmann wrote: Excerpts from Gorka Eguileor's message of 2015-07-28 10:37:42 +0200: On Fri, Jul 24, 2015 at 10:08:45AM -0400, Doug Hellmann wrote: Excerpts from Kekane, Abhishek's message of 2015-07-24 06:33:00 +: Hi Devs, X-Openstack-Request-Id. We have analysed python-cinderclient, python-glanceclient, python-novaclient, python-keystoneclient and python-neutronclient to check the return types. There are 9 ways return values are returned from python clients: 1. List 2. Dict 3. Resource class object 4. None 5. Tuple 6. Exception 7. Boolean (True/False, for keystoneclient) 8. Generator (for list api's in glanceclient) 9. String (for novaclient) Out of 9 we have solution for all return types except generator. In case of glance-client list api's are returning generator which is immutable. So it is not possible to return request-id in this case, which is a blocker for adopting the solution. I have added detail analysis for above return types in etherpad [2] as solution #3. If you have any suggestion in case of generation type then please let me know. It should be possible to create a new class to wrap the existing generator and implement the iterator protocol [3]. [3] https://docs.python.org/2/reference/expressions.html#generator-iterator-methods Doug Unless I'm missing something I think we wouldn't even need to create a new class that implements the iterator protocol, we can just return a generator that generates from the other one. For example, for each of the requests, if we get the generator in variable *result* that returns dictionaries and we want to add *headers* to each dictionary: return (DictWithHeaders(resource, headers) for resource in result) Wouldn't that work? That would work, but it wouldn't be consistent with the way I read [2] as describing how the other methods are being updated. For example, a method that now returns a list() will return a ListWithHeaders(), and only that returned object will have the headers, not its contents. A caller could do: You are right, it wouldn't be consistent with other methods, and that's not good. So the new iterator class wrapper seems to be the way to go. Gorka. response = client.some_method_returning_a_list() print reponse.headers but could not do print response[0].headers and get the same values. Creating a GeneratorWithHeaders class mirrors that behavior for methods that return generators instead of lists. Doug [2] https://etherpad.openstack.org/p/request-id __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] cross project communication: Return request-id to caller
On Fri, Jul 24, 2015 at 10:08:45AM -0400, Doug Hellmann wrote: Excerpts from Kekane, Abhishek's message of 2015-07-24 06:33:00 +: Hi Devs, X-Openstack-Request-Id. We have analysed python-cinderclient, python-glanceclient, python-novaclient, python-keystoneclient and python-neutronclient to check the return types. There are 9 ways return values are returned from python clients: 1. List 2. Dict 3. Resource class object 4. None 5. Tuple 6. Exception 7. Boolean (True/False, for keystoneclient) 8. Generator (for list api's in glanceclient) 9. String (for novaclient) Out of 9 we have solution for all return types except generator. In case of glance-client list api's are returning generator which is immutable. So it is not possible to return request-id in this case, which is a blocker for adopting the solution. I have added detail analysis for above return types in etherpad [2] as solution #3. If you have any suggestion in case of generation type then please let me know. It should be possible to create a new class to wrap the existing generator and implement the iterator protocol [3]. [3] https://docs.python.org/2/reference/expressions.html#generator-iterator-methods Doug Unless I'm missing something I think we wouldn't even need to create a new class that implements the iterator protocol, we can just return a generator that generates from the other one. For example, for each of the requests, if we get the generator in variable *result* that returns dictionaries and we want to add *headers* to each dictionary: return (DictWithHeaders(resource, headers) for resource in result) Wouldn't that work? Cheers, Gorka. [1] http://eavesdrop.openstack.org/meetings/crossproject/2015/crossproject.2015-07-07-21.01.log.html [2] https://etherpad.openstack.org/p/request-id Thanks Best Regards, Abhishek Kekane __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] cross project communication: Return request-id to caller
Excerpts from Gorka Eguileor's message of 2015-07-28 10:37:42 +0200: On Fri, Jul 24, 2015 at 10:08:45AM -0400, Doug Hellmann wrote: Excerpts from Kekane, Abhishek's message of 2015-07-24 06:33:00 +: Hi Devs, X-Openstack-Request-Id. We have analysed python-cinderclient, python-glanceclient, python-novaclient, python-keystoneclient and python-neutronclient to check the return types. There are 9 ways return values are returned from python clients: 1. List 2. Dict 3. Resource class object 4. None 5. Tuple 6. Exception 7. Boolean (True/False, for keystoneclient) 8. Generator (for list api's in glanceclient) 9. String (for novaclient) Out of 9 we have solution for all return types except generator. In case of glance-client list api's are returning generator which is immutable. So it is not possible to return request-id in this case, which is a blocker for adopting the solution. I have added detail analysis for above return types in etherpad [2] as solution #3. If you have any suggestion in case of generation type then please let me know. It should be possible to create a new class to wrap the existing generator and implement the iterator protocol [3]. [3] https://docs.python.org/2/reference/expressions.html#generator-iterator-methods Doug Unless I'm missing something I think we wouldn't even need to create a new class that implements the iterator protocol, we can just return a generator that generates from the other one. For example, for each of the requests, if we get the generator in variable *result* that returns dictionaries and we want to add *headers* to each dictionary: return (DictWithHeaders(resource, headers) for resource in result) Wouldn't that work? That would work, but it wouldn't be consistent with the way I read [2] as describing how the other methods are being updated. For example, a method that now returns a list() will return a ListWithHeaders(), and only that returned object will have the headers, not its contents. A caller could do: response = client.some_method_returning_a_list() print reponse.headers but could not do print response[0].headers and get the same values. Creating a GeneratorWithHeaders class mirrors that behavior for methods that return generators instead of lists. Doug [2] https://etherpad.openstack.org/p/request-id __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [all] cross project communication: Return request-id to caller
Hi Devs, X-Openstack-Request-Id. We have analysed python-cinderclient, python-glanceclient, python-novaclient, python-keystoneclient and python-neutronclient to check the return types. There are 9 ways return values are returned from python clients: 1. List 2. Dict 3. Resource class object 4. None 5. Tuple 6. Exception 7. Boolean (True/False, for keystoneclient) 8. Generator (for list api's in glanceclient) 9. String (for novaclient) Out of 9 we have solution for all return types except generator. In case of glance-client list api's are returning generator which is immutable. So it is not possible to return request-id in this case, which is a blocker for adopting the solution. I have added detail analysis for above return types in etherpad [2] as solution #3. If you have any suggestion in case of generation type then please let me know. [1] http://eavesdrop.openstack.org/meetings/crossproject/2015/crossproject.2015-07-07-21.01.log.html [2] https://etherpad.openstack.org/p/request-id Thanks Best Regards, Abhishek Kekane __ Disclaimer: This email and any attachments are sent in strictest confidence for the sole use of the addressee and may contain legally privileged, confidential, and proprietary data. If you are not the intended recipient, please advise the sender by replying promptly to this email and then delete and destroy this email and any attachments without any further use, copying or forwarding.__ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] cross project communication: Return request-id to caller
Excerpts from Kekane, Abhishek's message of 2015-07-24 06:33:00 +: Hi Devs, X-Openstack-Request-Id. We have analysed python-cinderclient, python-glanceclient, python-novaclient, python-keystoneclient and python-neutronclient to check the return types. There are 9 ways return values are returned from python clients: 1. List 2. Dict 3. Resource class object 4. None 5. Tuple 6. Exception 7. Boolean (True/False, for keystoneclient) 8. Generator (for list api's in glanceclient) 9. String (for novaclient) Out of 9 we have solution for all return types except generator. In case of glance-client list api's are returning generator which is immutable. So it is not possible to return request-id in this case, which is a blocker for adopting the solution. I have added detail analysis for above return types in etherpad [2] as solution #3. If you have any suggestion in case of generation type then please let me know. It should be possible to create a new class to wrap the existing generator and implement the iterator protocol [3]. [3] https://docs.python.org/2/reference/expressions.html#generator-iterator-methods Doug [1] http://eavesdrop.openstack.org/meetings/crossproject/2015/crossproject.2015-07-07-21.01.log.html [2] https://etherpad.openstack.org/p/request-id Thanks Best Regards, Abhishek Kekane __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] cross project communication: Return request-id to caller
On Wednesday 3 June 2015 at 12:13, Miguel Ángel Ajo wrote: Doesn’t this overlap with the work done for the OSProfiler ? More comments inline. Miguel Ángel Ajo On Wednesday, 3 de June de 2015 at 11:43, Kekane, Abhishek wrote: Hi Devs, So for I have got following responses on the proposed solutions: Solution 1: Return tuple containing headers and body from - 3 +1 Solution 2: Use thread local storage to store 'x-openstack-request-id' returned from headers - 0 +1 Solution 3: Unique request-id across OpenStack Services - 1 +1 I’d vote for Solution 3, without involving keystone (first caller with no req-id generates one randomly), the req-id contains a call/hop count, which is incremented on every new call... sorry, that suggestion is naive, simply incrementing the call count won’t work as calls diverge we will have duplicate req-id + hop’s Requesting community people, cross-project members and PTL's to go through this mailing thread [1] and give your suggestions/opinions about the solutions proposed so that It will be easy to finalize the solution. [1] http://lists.openstack.org/pipermail/openstack-dev/2015-May/064842.html Thanks Regards, Abhishek Kekane -Original Message- From: Nikhil Komawar [mailto:nik.koma...@gmail.com] Sent: 28 May 2015 12:34 To: openstack-dev@lists.openstack.org (mailto:openstack-dev@lists.openstack.org) Subject: Re: [openstack-dev] [all] cross project communication: Return request-id to caller Did you get to talk with anyone in the LogWG ( https://wiki.openstack.org/wiki/LogWorkingGroup )? In wonder what kind of recommendations, standards we can come up with while adopting a cross project solution. If our logs follow certain prefix and or suffix style across projects, that would help a long way. Personally: +1 on Solution 1 On 5/28/15 2:14 AM, Kekane, Abhishek wrote: Hi Devs, Thank you for your opinions/thoughts. However I would like to suggest that please give +1 against the solution which you will like to propose so that at the end it will be helpful for us to consolidate the voting against each solution and make some decision. Thanks in advance. Abhishek Kekane *From:*Joe Gordon [mailto:joe.gord...@gmail.com] *Sent:* 28 May 2015 00:31 *To:* OpenStack Development Mailing List (not for usage questions) *Subject:* Re: [openstack-dev] [all] cross project communication: Return request-id to caller On Wed, May 27, 2015 at 12:06 AM, Kekane, Abhishek abhishek.kek...@nttdata.com (mailto:abhishek.kek...@nttdata.com) mailto:abhishek.kek...@nttdata.com wrote: Hi Devs, Each OpenStack service sends a request ID header with HTTP responses. This request ID can be useful for tracking down problems in the logs. However, when operation crosses service boundaries, this tracking can become difficult, as each service has its own request ID. Request ID is not returned to the caller, so it is not easy to track the request. This becomes especially problematic when requests are coming in parallel. For example, glance will call cinder for creating image, but that cinder instance may be handling several other requests at the same time. By using same request ID in the log, user can easily find the cinder request ID that is same as glance request ID in the g-api log. It will help operators/developers to analyse logs effectively. Thank you for writing this up. To address this issue we have come up with following solutions: Solution 1: Return tuple containing headers and body from respective clients (also favoured by Joe Gordon) Reference: https://review.openstack.org/#/c/156508/6/specs/log-request-id-mapping s.rst Pros: 1. Maintains backward compatibility 2. Effective debugging/analysing of the problem as both calling service request-id and called service request-id are logged in same log message 3. Build a full call graph 4. End user will able to know the request-id of the request and can approach service provider to know the cause of failure of particular request. Cons: 1. The changes need to be done first in cross-projects before making changes in clients 2. Applications which are using python-*clients needs to do required changes (check return type of response) Additional cons: 3. Cannot simply search all logs (ala logstash) using the request-id returned to the user without any post processing of the logs. Solution 2: Use thread local storage to store 'x-openstack-request-id' returned from
Re: [openstack-dev] [all] cross project communication: Return request-id to caller
Hi Devs, So for I have got following responses on the proposed solutions: Solution 1: Return tuple containing headers and body from - 3 +1 Solution 2: Use thread local storage to store 'x-openstack-request-id' returned from headers - 0 +1 Solution 3: Unique request-id across OpenStack Services - 1 +1 Requesting community people, cross-project members and PTL's to go through this mailing thread [1] and give your suggestions/opinions about the solutions proposed so that It will be easy to finalize the solution. [1] http://lists.openstack.org/pipermail/openstack-dev/2015-May/064842.html Thanks Regards, Abhishek Kekane -Original Message- From: Nikhil Komawar [mailto:nik.koma...@gmail.com] Sent: 28 May 2015 12:34 To: openstack-dev@lists.openstack.org Subject: Re: [openstack-dev] [all] cross project communication: Return request-id to caller Did you get to talk with anyone in the LogWG ( https://wiki.openstack.org/wiki/LogWorkingGroup )? In wonder what kind of recommendations, standards we can come up with while adopting a cross project solution. If our logs follow certain prefix and or suffix style across projects, that would help a long way. Personally: +1 on Solution 1 On 5/28/15 2:14 AM, Kekane, Abhishek wrote: Hi Devs, Thank you for your opinions/thoughts. However I would like to suggest that please give +1 against the solution which you will like to propose so that at the end it will be helpful for us to consolidate the voting against each solution and make some decision. Thanks in advance. Abhishek Kekane *From:*Joe Gordon [mailto:joe.gord...@gmail.com] *Sent:* 28 May 2015 00:31 *To:* OpenStack Development Mailing List (not for usage questions) *Subject:* Re: [openstack-dev] [all] cross project communication: Return request-id to caller On Wed, May 27, 2015 at 12:06 AM, Kekane, Abhishek abhishek.kek...@nttdata.com mailto:abhishek.kek...@nttdata.com wrote: Hi Devs, Each OpenStack service sends a request ID header with HTTP responses. This request ID can be useful for tracking down problems in the logs. However, when operation crosses service boundaries, this tracking can become difficult, as each service has its own request ID. Request ID is not returned to the caller, so it is not easy to track the request. This becomes especially problematic when requests are coming in parallel. For example, glance will call cinder for creating image, but that cinder instance may be handling several other requests at the same time. By using same request ID in the log, user can easily find the cinder request ID that is same as glance request ID in the g-api log. It will help operators/developers to analyse logs effectively. Thank you for writing this up. To address this issue we have come up with following solutions: Solution 1: Return tuple containing headers and body from respective clients (also favoured by Joe Gordon) Reference: https://review.openstack.org/#/c/156508/6/specs/log-request-id-mapping s.rst Pros: 1. Maintains backward compatibility 2. Effective debugging/analysing of the problem as both calling service request-id and called service request-id are logged in same log message 3. Build a full call graph 4. End user will able to know the request-id of the request and can approach service provider to know the cause of failure of particular request. Cons: 1. The changes need to be done first in cross-projects before making changes in clients 2. Applications which are using python-*clients needs to do required changes (check return type of response) Additional cons: 3. Cannot simply search all logs (ala logstash) using the request-id returned to the user without any post processing of the logs. Solution 2: Use thread local storage to store 'x-openstack-request-id' returned from headers (suggested by Doug Hellmann) Reference: https://review.openstack.org/#/c/156508/9/specs/log-request-id-mapping s.rst Add new method 'get_openstack_request_id' to return this request-id to the caller. Pros: 1. Doesn't break compatibility 2. Minimal changes are required in client 3. Build a full call graph Cons: 1. Malicious user can send long request-id to fill up the disk-space, resulting in potential DoS 2. Changes need to be done in all python-*clients 3. Last request id should be flushed out in a subsequent call otherwise it will return wrong request id to the caller Solution 3: Unique request-id across OpenStack Services (suggested by Jamie Lennox) Reference: https://review.openstack.org/#/c/156508/10/specs/log-request-id-mappin gs.rst Get 'x-openstack
Re: [openstack-dev] [all] cross project communication: Return request-id to caller
Doesn’t this overlap with the work done for the OSProfiler ? More comments inline. Miguel Ángel Ajo On Wednesday, 3 de June de 2015 at 11:43, Kekane, Abhishek wrote: Hi Devs, So for I have got following responses on the proposed solutions: Solution 1: Return tuple containing headers and body from - 3 +1 Solution 2: Use thread local storage to store 'x-openstack-request-id' returned from headers - 0 +1 Solution 3: Unique request-id across OpenStack Services - 1 +1 I’d vote for Solution 3, without involving keystone (first caller with no req-id generates one randomly), the req-id contains a call/hop count, which is incremented on every new call... Requesting community people, cross-project members and PTL's to go through this mailing thread [1] and give your suggestions/opinions about the solutions proposed so that It will be easy to finalize the solution. [1] http://lists.openstack.org/pipermail/openstack-dev/2015-May/064842.html Thanks Regards, Abhishek Kekane -Original Message- From: Nikhil Komawar [mailto:nik.koma...@gmail.com] Sent: 28 May 2015 12:34 To: openstack-dev@lists.openstack.org (mailto:openstack-dev@lists.openstack.org) Subject: Re: [openstack-dev] [all] cross project communication: Return request-id to caller Did you get to talk with anyone in the LogWG ( https://wiki.openstack.org/wiki/LogWorkingGroup )? In wonder what kind of recommendations, standards we can come up with while adopting a cross project solution. If our logs follow certain prefix and or suffix style across projects, that would help a long way. Personally: +1 on Solution 1 On 5/28/15 2:14 AM, Kekane, Abhishek wrote: Hi Devs, Thank you for your opinions/thoughts. However I would like to suggest that please give +1 against the solution which you will like to propose so that at the end it will be helpful for us to consolidate the voting against each solution and make some decision. Thanks in advance. Abhishek Kekane *From:*Joe Gordon [mailto:joe.gord...@gmail.com] *Sent:* 28 May 2015 00:31 *To:* OpenStack Development Mailing List (not for usage questions) *Subject:* Re: [openstack-dev] [all] cross project communication: Return request-id to caller On Wed, May 27, 2015 at 12:06 AM, Kekane, Abhishek abhishek.kek...@nttdata.com mailto:abhishek.kek...@nttdata.com wrote: Hi Devs, Each OpenStack service sends a request ID header with HTTP responses. This request ID can be useful for tracking down problems in the logs. However, when operation crosses service boundaries, this tracking can become difficult, as each service has its own request ID. Request ID is not returned to the caller, so it is not easy to track the request. This becomes especially problematic when requests are coming in parallel. For example, glance will call cinder for creating image, but that cinder instance may be handling several other requests at the same time. By using same request ID in the log, user can easily find the cinder request ID that is same as glance request ID in the g-api log. It will help operators/developers to analyse logs effectively. Thank you for writing this up. To address this issue we have come up with following solutions: Solution 1: Return tuple containing headers and body from respective clients (also favoured by Joe Gordon) Reference: https://review.openstack.org/#/c/156508/6/specs/log-request-id-mapping s.rst Pros: 1. Maintains backward compatibility 2. Effective debugging/analysing of the problem as both calling service request-id and called service request-id are logged in same log message 3. Build a full call graph 4. End user will able to know the request-id of the request and can approach service provider to know the cause of failure of particular request. Cons: 1. The changes need to be done first in cross-projects before making changes in clients 2. Applications which are using python-*clients needs to do required changes (check return type of response) Additional cons: 3. Cannot simply search all logs (ala logstash) using the request-id returned to the user without any post processing of the logs. Solution 2: Use thread local storage to store 'x-openstack-request-id' returned from headers (suggested by Doug Hellmann) Reference: https://review.openstack.org/#/c/156508/9/specs/log-request-id-mapping s.rst Add new method 'get_openstack_request_id' to return this request-id to the caller. Pros: 1. Doesn't break compatibility 2. Minimal changes are required in client 3. Build a full call graph Cons: 1. Malicious user can send long request-id
Re: [openstack-dev] [all] cross project communication: Return request-id to caller
Hi Nikhil, Thanks for the opinion, actually I was in different time-zone and was not aware about this LogWG. I will certainly approach them in next meeting. Thank you, Abhishek Kekane -Original Message- From: Nikhil Komawar [mailto:nik.koma...@gmail.com] Sent: 28 May 2015 12:34 To: openstack-dev@lists.openstack.org Subject: Re: [openstack-dev] [all] cross project communication: Return request-id to caller Did you get to talk with anyone in the LogWG ( https://wiki.openstack.org/wiki/LogWorkingGroup )? In wonder what kind of recommendations, standards we can come up with while adopting a cross project solution. If our logs follow certain prefix and or suffix style across projects, that would help a long way. Personally: +1 on Solution 1 On 5/28/15 2:14 AM, Kekane, Abhishek wrote: Hi Devs, Thank you for your opinions/thoughts. However I would like to suggest that please give +1 against the solution which you will like to propose so that at the end it will be helpful for us to consolidate the voting against each solution and make some decision. Thanks in advance. Abhishek Kekane *From:*Joe Gordon [mailto:joe.gord...@gmail.com] *Sent:* 28 May 2015 00:31 *To:* OpenStack Development Mailing List (not for usage questions) *Subject:* Re: [openstack-dev] [all] cross project communication: Return request-id to caller On Wed, May 27, 2015 at 12:06 AM, Kekane, Abhishek abhishek.kek...@nttdata.com mailto:abhishek.kek...@nttdata.com wrote: Hi Devs, Each OpenStack service sends a request ID header with HTTP responses. This request ID can be useful for tracking down problems in the logs. However, when operation crosses service boundaries, this tracking can become difficult, as each service has its own request ID. Request ID is not returned to the caller, so it is not easy to track the request. This becomes especially problematic when requests are coming in parallel. For example, glance will call cinder for creating image, but that cinder instance may be handling several other requests at the same time. By using same request ID in the log, user can easily find the cinder request ID that is same as glance request ID in the g-api log. It will help operators/developers to analyse logs effectively. Thank you for writing this up. To address this issue we have come up with following solutions: Solution 1: Return tuple containing headers and body from respective clients (also favoured by Joe Gordon) Reference: https://review.openstack.org/#/c/156508/6/specs/log-request-id-mapping s.rst Pros: 1. Maintains backward compatibility 2. Effective debugging/analysing of the problem as both calling service request-id and called service request-id are logged in same log message 3. Build a full call graph 4. End user will able to know the request-id of the request and can approach service provider to know the cause of failure of particular request. Cons: 1. The changes need to be done first in cross-projects before making changes in clients 2. Applications which are using python-*clients needs to do required changes (check return type of response) Additional cons: 3. Cannot simply search all logs (ala logstash) using the request-id returned to the user without any post processing of the logs. Solution 2: Use thread local storage to store 'x-openstack-request-id' returned from headers (suggested by Doug Hellmann) Reference: https://review.openstack.org/#/c/156508/9/specs/log-request-id-mapping s.rst Add new method 'get_openstack_request_id' to return this request-id to the caller. Pros: 1. Doesn't break compatibility 2. Minimal changes are required in client 3. Build a full call graph Cons: 1. Malicious user can send long request-id to fill up the disk-space, resulting in potential DoS 2. Changes need to be done in all python-*clients 3. Last request id should be flushed out in a subsequent call otherwise it will return wrong request id to the caller Solution 3: Unique request-id across OpenStack Services (suggested by Jamie Lennox) Reference: https://review.openstack.org/#/c/156508/10/specs/log-request-id-mappin gs.rst Get 'x-openstack-request-id' from auth plugin and add it to the request headers. If 'x-openstack-request-id' key is present in the request header, then it will use the same one further or else it will generate a new one. Dependencies: https://review.openstack.org/#/c/164582/ - Include request-id in auth plugin and add it to request headers https://review.openstack.org/#/c/166063/ - Add session-object for glance
Re: [openstack-dev] [all] cross project communication: Return request-id to caller
Hi Devs, Thank you for your opinions/thoughts. However I would like to suggest that please give +1 against the solution which you will like to propose so that at the end it will be helpful for us to consolidate the voting against each solution and make some decision. Thanks in advance. Abhishek Kekane From: Joe Gordon [mailto:joe.gord...@gmail.com] Sent: 28 May 2015 00:31 To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [all] cross project communication: Return request-id to caller On Wed, May 27, 2015 at 12:06 AM, Kekane, Abhishek abhishek.kek...@nttdata.commailto:abhishek.kek...@nttdata.com wrote: Hi Devs, Each OpenStack service sends a request ID header with HTTP responses. This request ID can be useful for tracking down problems in the logs. However, when operation crosses service boundaries, this tracking can become difficult, as each service has its own request ID. Request ID is not returned to the caller, so it is not easy to track the request. This becomes especially problematic when requests are coming in parallel. For example, glance will call cinder for creating image, but that cinder instance may be handling several other requests at the same time. By using same request ID in the log, user can easily find the cinder request ID that is same as glance request ID in the g-api log. It will help operators/developers to analyse logs effectively. Thank you for writing this up. To address this issue we have come up with following solutions: Solution 1: Return tuple containing headers and body from respective clients (also favoured by Joe Gordon) Reference: https://review.openstack.org/#/c/156508/6/specs/log-request-id-mappings.rst Pros: 1. Maintains backward compatibility 2. Effective debugging/analysing of the problem as both calling service request-id and called service request-id are logged in same log message 3. Build a full call graph 4. End user will able to know the request-id of the request and can approach service provider to know the cause of failure of particular request. Cons: 1. The changes need to be done first in cross-projects before making changes in clients 2. Applications which are using python-*clients needs to do required changes (check return type of response) Additional cons: 3. Cannot simply search all logs (ala logstash) using the request-id returned to the user without any post processing of the logs. Solution 2: Use thread local storage to store 'x-openstack-request-id' returned from headers (suggested by Doug Hellmann) Reference: https://review.openstack.org/#/c/156508/9/specs/log-request-id-mappings.rst Add new method ‘get_openstack_request_id’ to return this request-id to the caller. Pros: 1. Doesn’t break compatibility 2. Minimal changes are required in client 3. Build a full call graph Cons: 1. Malicious user can send long request-id to fill up the disk-space, resulting in potential DoS 2. Changes need to be done in all python-*clients 3. Last request id should be flushed out in a subsequent call otherwise it will return wrong request id to the caller Solution 3: Unique request-id across OpenStack Services (suggested by Jamie Lennox) Reference: https://review.openstack.org/#/c/156508/10/specs/log-request-id-mappings.rst Get 'x-openstack-request-id' from auth plugin and add it to the request headers. If 'x-openstack-request-id' key is present in the request header, then it will use the same one further or else it will generate a new one. Dependencies: https://review.openstack.org/#/c/164582/ - Include request-id in auth plugin and add it to request headers https://review.openstack.org/#/c/166063/ - Add session-object for glance client Add 'UserAuthPlugin' and '_ContextAuthPlugin' same as nova in cinder and neutron Pros: 1. Using same request id for the request crossing multiple service boundaries will help operators/developers identify the problem quickly 2. Required changes only in keystonemiddleware and oslo_middleware libraries. No changes are required in the python client bindings or OpenStack core services Cons: 1. As 'x-openstack-request-id' in the request header will be visible to the user, it is possible to send same request id for multiple requests which in turn could create more problems in case of troubleshooting cause of the failure as request_id middleware will not check for its uniqueness in the scope of the running OpenStack service. 2. Having the same request ID for all services for a single user API call means you cannot generate a full call graph. For example if a single user's nova API call produces 2 calls to glance you want to be able to differentiate the two different calls. During the Liberty design summit, I had a chance of discussing these designs with some of the core members like Doug, Joe Gordon, Jamie Lennox etc. But not able to came to any conclusion on the final design and know the communities direction by which way
Re: [openstack-dev] [all] cross project communication: Return request-id to caller
Did you get to talk with anyone in the LogWG ( https://wiki.openstack.org/wiki/LogWorkingGroup )? In wonder what kind of recommendations, standards we can come up with while adopting a cross project solution. If our logs follow certain prefix and or suffix style across projects, that would help a long way. Personally: +1 on Solution 1 On 5/28/15 2:14 AM, Kekane, Abhishek wrote: Hi Devs, Thank you for your opinions/thoughts. However I would like to suggest that please give +1 against the solution which you will like to propose so that at the end it will be helpful for us to consolidate the voting against each solution and make some decision. Thanks in advance. Abhishek Kekane *From:*Joe Gordon [mailto:joe.gord...@gmail.com] *Sent:* 28 May 2015 00:31 *To:* OpenStack Development Mailing List (not for usage questions) *Subject:* Re: [openstack-dev] [all] cross project communication: Return request-id to caller On Wed, May 27, 2015 at 12:06 AM, Kekane, Abhishek abhishek.kek...@nttdata.com mailto:abhishek.kek...@nttdata.com wrote: Hi Devs, Each OpenStack service sends a request ID header with HTTP responses. This request ID can be useful for tracking down problems in the logs. However, when operation crosses service boundaries, this tracking can become difficult, as each service has its own request ID. Request ID is not returned to the caller, so it is not easy to track the request. This becomes especially problematic when requests are coming in parallel. For example, glance will call cinder for creating image, but that cinder instance may be handling several other requests at the same time. By using same request ID in the log, user can easily find the cinder request ID that is same as glance request ID in the g-api log. It will help operators/developers to analyse logs effectively. Thank you for writing this up. To address this issue we have come up with following solutions: Solution 1: Return tuple containing headers and body from respective clients (also favoured by Joe Gordon) Reference: https://review.openstack.org/#/c/156508/6/specs/log-request-id-mappings.rst Pros: 1. Maintains backward compatibility 2. Effective debugging/analysing of the problem as both calling service request-id and called service request-id are logged in same log message 3. Build a full call graph 4. End user will able to know the request-id of the request and can approach service provider to know the cause of failure of particular request. Cons: 1. The changes need to be done first in cross-projects before making changes in clients 2. Applications which are using python-*clients needs to do required changes (check return type of response) Additional cons: 3. Cannot simply search all logs (ala logstash) using the request-id returned to the user without any post processing of the logs. Solution 2: Use thread local storage to store 'x-openstack-request-id' returned from headers (suggested by Doug Hellmann) Reference: https://review.openstack.org/#/c/156508/9/specs/log-request-id-mappings.rst Add new method ‘get_openstack_request_id’ to return this request-id to the caller. Pros: 1. Doesn’t break compatibility 2. Minimal changes are required in client 3. Build a full call graph Cons: 1. Malicious user can send long request-id to fill up the disk-space, resulting in potential DoS 2. Changes need to be done in all python-*clients 3. Last request id should be flushed out in a subsequent call otherwise it will return wrong request id to the caller Solution 3: Unique request-id across OpenStack Services (suggested by Jamie Lennox) Reference: https://review.openstack.org/#/c/156508/10/specs/log-request-id-mappings.rst Get 'x-openstack-request-id' from auth plugin and add it to the request headers. If 'x-openstack-request-id' key is present in the request header, then it will use the same one further or else it will generate a new one. Dependencies: https://review.openstack.org/#/c/164582/ - Include request-id in auth plugin and add it to request headers https://review.openstack.org/#/c/166063/ - Add session-object for glance client Add 'UserAuthPlugin' and '_ContextAuthPlugin' same as nova in cinder and neutron Pros: 1. Using same request id for the request crossing multiple service boundaries will help operators/developers identify the problem quickly 2. Required changes only in keystonemiddleware and oslo_middleware libraries. No changes are required in the python client bindings or OpenStack core
Re: [openstack-dev] [all] cross project communication: Return request-id to caller
On Wed, May 27, 2015 at 10:31 AM, Mike Bayer mba...@redhat.com wrote: On 5/27/15 3:06 AM, Kekane, Abhishek wrote: Hi Devs, Each OpenStack service sends a request ID header with HTTP responses. This request ID can be useful for tracking down problems in the logs. However, when operation crosses service boundaries, this tracking can become difficult, as each service has its own request ID. Request ID is not returned to the caller, so it is not easy to track the request. This becomes especially problematic when requests are coming in parallel. For example, glance will call cinder for creating image, but that cinder instance may be handling several other requests at the same time. By using same request ID in the log, user can easily find the cinder request ID that is same as glance request ID in the g-api log. It will help operators/developers to analyse logs effectively. To address this issue we have come up with following solutions: Solution 1: Return tuple containing headers and body from respective clients (also favoured by Joe Gordon) Reference: https://review.openstack.org/#/c/156508/6/specs/log-request-id-mappings.rst I like solution 1 as well as solution 3 at the same time, in fact. There's usefulness to being able to easily identify a set of requests as all part of the same operation as well as being able to identify a call's location in the hierarchy. In fact does solution #1 make the hierarchy apparent ? I'd want it to do that, e.g. if call A calls B, which calls C and D, I'd want to know that the dependency tree is A-B-(C, D), and not just a bucket of (A, B, C, D). #1 should make the hierarchy apparent. That IMHO is the biggest pro for #1. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] cross project communication: Return request-id to caller
On Wed, May 27, 2015 at 12:06 AM, Kekane, Abhishek abhishek.kek...@nttdata.com wrote: Hi Devs, Each OpenStack service sends a request ID header with HTTP responses. This request ID can be useful for tracking down problems in the logs. However, when operation crosses service boundaries, this tracking can become difficult, as each service has its own request ID. Request ID is not returned to the caller, so it is not easy to track the request. This becomes especially problematic when requests are coming in parallel. For example, glance will call cinder for creating image, but that cinder instance may be handling several other requests at the same time. By using same request ID in the log, user can easily find the cinder request ID that is same as glance request ID in the g-api log. It will help operators/developers to analyse logs effectively. Thank you for writing this up. To address this issue we have come up with following solutions: Solution 1: Return tuple containing headers and body from respective clients (also favoured by Joe Gordon) Reference: https://review.openstack.org/#/c/156508/6/specs/log-request-id-mappings.rst Pros: 1. Maintains backward compatibility 2. Effective debugging/analysing of the problem as both calling service request-id and called service request-id are logged in same log message 3. Build a full call graph 4. End user will able to know the request-id of the request and can approach service provider to know the cause of failure of particular request. Cons: 1. The changes need to be done first in cross-projects before making changes in clients 2. Applications which are using python-*clients needs to do required changes (check return type of response) Additional cons: 3. Cannot simply search all logs (ala logstash) using the request-id returned to the user without any post processing of the logs. Solution 2: Use thread local storage to store 'x-openstack-request-id' returned from headers (suggested by Doug Hellmann) Reference: https://review.openstack.org/#/c/156508/9/specs/log-request-id-mappings.rst Add new method ‘get_openstack_request_id’ to return this request-id to the caller. Pros: 1. Doesn’t break compatibility 2. Minimal changes are required in client 3. Build a full call graph Cons: 1. Malicious user can send long request-id to fill up the disk-space, resulting in potential DoS 2. Changes need to be done in all python-*clients 3. Last request id should be flushed out in a subsequent call otherwise it will return wrong request id to the caller Solution 3: Unique request-id across OpenStack Services (suggested by Jamie Lennox) Reference: https://review.openstack.org/#/c/156508/10/specs/log-request-id-mappings.rst Get 'x-openstack-request-id' from auth plugin and add it to the request headers. If 'x-openstack-request-id' key is present in the request header, then it will use the same one further or else it will generate a new one. Dependencies: https://review.openstack.org/#/c/164582/ - Include request-id in auth plugin and add it to request headers https://review.openstack.org/#/c/166063/ - Add session-object for glance client Add 'UserAuthPlugin' and '_ContextAuthPlugin' same as nova in cinder and neutron Pros: 1. Using same request id for the request crossing multiple service boundaries will help operators/developers identify the problem quickly 2. Required changes only in keystonemiddleware and oslo_middleware libraries. No changes are required in the python client bindings or OpenStack core services Cons: 1. As 'x-openstack-request-id' in the request header will be visible to the user, it is possible to send same request id for multiple requests which in turn could create more problems in case of troubleshooting cause of the failure as request_id middleware will not check for its uniqueness in the scope of the running OpenStack service. 2. Having the same request ID for all services for a single user API call means you cannot generate a full call graph. For example if a single user's nova API call produces 2 calls to glance you want to be able to differentiate the two different calls. During the Liberty design summit, I had a chance of discussing these designs with some of the core members like Doug, Joe Gordon, Jamie Lennox etc. But not able to came to any conclusion on the final design and know the communities direction by which way they want to use this request-id effectively. However IMO, solution 1 sounds more useful as the debugger can able to build the full call graph which can be helpful for analysing gate failures effectively as well as end user will be able to know his request-id and can track his request. I request all community members to go through these solutions and let us know which is the appropriate way to improve the logs by logging request-id. Thanks Regards,
Re: [openstack-dev] [all] cross project communication: Return request-id to caller
On 5/27/15, 10:15, Doug Hellmann d...@doughellmann.com wrote: Excerpts from Kekane, Abhishek's message of 2015-05-27 07:06:56 +: Hi Devs, Each OpenStack service sends a request ID header with HTTP responses. This request ID can be useful for tracking down problems in the logs. However, when operation crosses service boundaries, this tracking can become difficult, as each service has its own request ID. Request ID is not returned to the caller, so it is not easy to track the request. This becomes especially problematic when requests are coming in parallel. For example, glance will call cinder for creating image, but that cinder instance may be handling several other requests at the same time. By using same request ID in the log, user can easily find the cinder request ID that is same as glance request ID in the g-api log. It will help operators/developers to analyse logs effectively. To address this issue we have come up with following solutions: Solution 1: Return tuple containing headers and body from respective clients (also favoured by Joe Gordon) Reference: https://review.openstack.org/#/c/156508/6/specs/log-request-id-mappings.r st Pros: 1. Maintains backward compatibility 2. Effective debugging/analysing of the problem as both calling service request-id and called service request-id are logged in same log message 3. Build a full call graph 4. End user will able to know the request-id of the request and can approach service provider to know the cause of failure of particular request. Cons: 1. The changes need to be done first in cross-projects before making changes in clients 2. Applications which are using python-*clients needs to do required changes (check return type of response) Solution 2: Use thread local storage to store 'x-openstack-request-id' returned from headers (suggested by Doug Hellmann) Reference: https://review.openstack.org/#/c/156508/9/specs/log-request-id-mappings.r st Add new method 'get_openstack_request_id' to return this request-id to the caller. Pros: 1. Doesn't break compatibility 2. Minimal changes are required in client 3. Build a full call graph Cons: 1. Malicious user can send long request-id to fill up the disk-space, resulting in potential DoS 2. Changes need to be done in all python-*clients 3. Last request id should be flushed out in a subsequent call otherwise it will return wrong request id to the caller Solution 3: Unique request-id across OpenStack Services (suggested by Jamie Lennox) Reference: https://review.openstack.org/#/c/156508/10/specs/log-request-id-mappings. rst Get 'x-openstack-request-id' from auth plugin and add it to the request headers. If 'x-openstack-request-id' key is present in the request header, then it will use the same one further or else it will generate a new one. Dependencies: https://review.openstack.org/#/c/164582/ - Include request-id in auth plugin and add it to request headers https://review.openstack.org/#/c/166063/ - Add session-object for glance client Add 'UserAuthPlugin' and '_ContextAuthPlugin' same as nova in cinder and neutron Pros: 1. Using same request id for the request crossing multiple service boundaries will help operators/developers identify the problem quickly 2. Required changes only in keystonemiddleware and oslo_middleware libraries. No changes are required in the python client bindings or OpenStack core services Cons: 1. As 'x-openstack-request-id' in the request header will be visible to the user, it is possible to send same request id for multiple requests which in turn could create more problems in case of troubleshooting cause of the failure as request_id middleware will not check for its uniqueness in the scope of the running OpenStack service. 2. Having the same request ID for all services for a single user API call means you cannot generate a full call graph. For example if a single user's nova API call produces 2 calls to glance you want to be able to differentiate the two different calls. During the Liberty design summit, I had a chance of discussing these designs with some of the core members like Doug, Joe Gordon, Jamie Lennox etc. But not able to came to any conclusion on the final design and know the communities direction by which way they want to use this request-id effectively. However IMO, solution 1 sounds more useful as the debugger can able to build the full call graph which can be helpful for analysing gate failures effectively as well as end user will be able to know his request-id and can track his request. I request all community members to go through these solutions and let us know which is the appropriate way to improve the logs by logging request-id. Robert Collins pointed out that os-profiler is already tracking requests across REST calls. Does it use one of the proposed methods? Since os-profiler is seeing more adoption, could we combine efforts here? Doug Speaking with my
Re: [openstack-dev] [all] cross project communication: Return request-id to caller
Excerpts from Kekane, Abhishek's message of 2015-05-27 07:06:56 +: Hi Devs, Each OpenStack service sends a request ID header with HTTP responses. This request ID can be useful for tracking down problems in the logs. However, when operation crosses service boundaries, this tracking can become difficult, as each service has its own request ID. Request ID is not returned to the caller, so it is not easy to track the request. This becomes especially problematic when requests are coming in parallel. For example, glance will call cinder for creating image, but that cinder instance may be handling several other requests at the same time. By using same request ID in the log, user can easily find the cinder request ID that is same as glance request ID in the g-api log. It will help operators/developers to analyse logs effectively. To address this issue we have come up with following solutions: Solution 1: Return tuple containing headers and body from respective clients (also favoured by Joe Gordon) Reference: https://review.openstack.org/#/c/156508/6/specs/log-request-id-mappings.rst Pros: 1. Maintains backward compatibility 2. Effective debugging/analysing of the problem as both calling service request-id and called service request-id are logged in same log message 3. Build a full call graph 4. End user will able to know the request-id of the request and can approach service provider to know the cause of failure of particular request. Cons: 1. The changes need to be done first in cross-projects before making changes in clients 2. Applications which are using python-*clients needs to do required changes (check return type of response) Solution 2: Use thread local storage to store 'x-openstack-request-id' returned from headers (suggested by Doug Hellmann) Reference: https://review.openstack.org/#/c/156508/9/specs/log-request-id-mappings.rst Add new method 'get_openstack_request_id' to return this request-id to the caller. Pros: 1. Doesn't break compatibility 2. Minimal changes are required in client 3. Build a full call graph Cons: 1. Malicious user can send long request-id to fill up the disk-space, resulting in potential DoS 2. Changes need to be done in all python-*clients 3. Last request id should be flushed out in a subsequent call otherwise it will return wrong request id to the caller Solution 3: Unique request-id across OpenStack Services (suggested by Jamie Lennox) Reference: https://review.openstack.org/#/c/156508/10/specs/log-request-id-mappings.rst Get 'x-openstack-request-id' from auth plugin and add it to the request headers. If 'x-openstack-request-id' key is present in the request header, then it will use the same one further or else it will generate a new one. Dependencies: https://review.openstack.org/#/c/164582/ - Include request-id in auth plugin and add it to request headers https://review.openstack.org/#/c/166063/ - Add session-object for glance client Add 'UserAuthPlugin' and '_ContextAuthPlugin' same as nova in cinder and neutron Pros: 1. Using same request id for the request crossing multiple service boundaries will help operators/developers identify the problem quickly 2. Required changes only in keystonemiddleware and oslo_middleware libraries. No changes are required in the python client bindings or OpenStack core services Cons: 1. As 'x-openstack-request-id' in the request header will be visible to the user, it is possible to send same request id for multiple requests which in turn could create more problems in case of troubleshooting cause of the failure as request_id middleware will not check for its uniqueness in the scope of the running OpenStack service. 2. Having the same request ID for all services for a single user API call means you cannot generate a full call graph. For example if a single user's nova API call produces 2 calls to glance you want to be able to differentiate the two different calls. During the Liberty design summit, I had a chance of discussing these designs with some of the core members like Doug, Joe Gordon, Jamie Lennox etc. But not able to came to any conclusion on the final design and know the communities direction by which way they want to use this request-id effectively. However IMO, solution 1 sounds more useful as the debugger can able to build the full call graph which can be helpful for analysing gate failures effectively as well as end user will be able to know his request-id and can track his request. I request all community members to go through these solutions and let us know which is the appropriate way to improve the logs by logging request-id. Robert Collins pointed out that os-profiler is already tracking requests across REST calls. Does it use one of the proposed methods? Since os-profiler is seeing more adoption, could we combine efforts here? Doug
Re: [openstack-dev] [all] cross project communication: Return request-id to caller
On 5/27/15 3:06 AM, Kekane, Abhishek wrote: Hi Devs, Each OpenStack service sends a request ID header with HTTP responses. This request ID can be useful for tracking down problems in the logs. However, when operation crosses service boundaries, this tracking can become difficult, as each service has its own request ID. Request ID is not returned to the caller, so it is not easy to track the request. This becomes especially problematic when requests are coming in parallel. For example, glance will call cinder for creating image, but that cinder instance may be handling several other requests at the same time. By using same request ID in the log, user can easily find the cinder request ID that is same as glance request ID in the g-api log. It will help operators/developers to analyse logs effectively. To address this issue we have come up with following solutions: Solution 1: Return tuple containing headers and body from respective clients (also favoured by Joe Gordon) Reference: https://review.openstack.org/#/c/156508/6/specs/log-request-id-mappings.rst I like solution 1 as well as solution 3 at the same time, in fact. There's usefulness to being able to easily identify a set of requests as all part of the same operation as well as being able to identify a call's location in the hierarchy. In fact does solution #1 make the hierarchy apparent ? I'd want it to do that, e.g. if call A calls B, which calls C and D, I'd want to know that the dependency tree is A-B-(C, D), and not just a bucket of (A, B, C, D). __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [all] cross project communication: Return request-id to caller
Hi Devs, Each OpenStack service sends a request ID header with HTTP responses. This request ID can be useful for tracking down problems in the logs. However, when operation crosses service boundaries, this tracking can become difficult, as each service has its own request ID. Request ID is not returned to the caller, so it is not easy to track the request. This becomes especially problematic when requests are coming in parallel. For example, glance will call cinder for creating image, but that cinder instance may be handling several other requests at the same time. By using same request ID in the log, user can easily find the cinder request ID that is same as glance request ID in the g-api log. It will help operators/developers to analyse logs effectively. To address this issue we have come up with following solutions: Solution 1: Return tuple containing headers and body from respective clients (also favoured by Joe Gordon) Reference: https://review.openstack.org/#/c/156508/6/specs/log-request-id-mappings.rst Pros: 1. Maintains backward compatibility 2. Effective debugging/analysing of the problem as both calling service request-id and called service request-id are logged in same log message 3. Build a full call graph 4. End user will able to know the request-id of the request and can approach service provider to know the cause of failure of particular request. Cons: 1. The changes need to be done first in cross-projects before making changes in clients 2. Applications which are using python-*clients needs to do required changes (check return type of response) Solution 2: Use thread local storage to store 'x-openstack-request-id' returned from headers (suggested by Doug Hellmann) Reference: https://review.openstack.org/#/c/156508/9/specs/log-request-id-mappings.rst Add new method 'get_openstack_request_id' to return this request-id to the caller. Pros: 1. Doesn't break compatibility 2. Minimal changes are required in client 3. Build a full call graph Cons: 1. Malicious user can send long request-id to fill up the disk-space, resulting in potential DoS 2. Changes need to be done in all python-*clients 3. Last request id should be flushed out in a subsequent call otherwise it will return wrong request id to the caller Solution 3: Unique request-id across OpenStack Services (suggested by Jamie Lennox) Reference: https://review.openstack.org/#/c/156508/10/specs/log-request-id-mappings.rst Get 'x-openstack-request-id' from auth plugin and add it to the request headers. If 'x-openstack-request-id' key is present in the request header, then it will use the same one further or else it will generate a new one. Dependencies: https://review.openstack.org/#/c/164582/ - Include request-id in auth plugin and add it to request headers https://review.openstack.org/#/c/166063/ - Add session-object for glance client Add 'UserAuthPlugin' and '_ContextAuthPlugin' same as nova in cinder and neutron Pros: 1. Using same request id for the request crossing multiple service boundaries will help operators/developers identify the problem quickly 2. Required changes only in keystonemiddleware and oslo_middleware libraries. No changes are required in the python client bindings or OpenStack core services Cons: 1. As 'x-openstack-request-id' in the request header will be visible to the user, it is possible to send same request id for multiple requests which in turn could create more problems in case of troubleshooting cause of the failure as request_id middleware will not check for its uniqueness in the scope of the running OpenStack service. 2. Having the same request ID for all services for a single user API call means you cannot generate a full call graph. For example if a single user's nova API call produces 2 calls to glance you want to be able to differentiate the two different calls. During the Liberty design summit, I had a chance of discussing these designs with some of the core members like Doug, Joe Gordon, Jamie Lennox etc. But not able to came to any conclusion on the final design and know the communities direction by which way they want to use this request-id effectively. However IMO, solution 1 sounds more useful as the debugger can able to build the full call graph which can be helpful for analysing gate failures effectively as well as end user will be able to know his request-id and can track his request. I request all community members to go through these solutions and let us know which is the appropriate way to improve the logs by logging request-id. Thanks Regards, Abhishek Kekane __ Disclaimer: This email and any attachments are sent in strictest confidence for the sole use of the addressee and may contain legally privileged, confidential, and proprietary data. If you are not the intended recipient, please advise the sender by replying promptly to this email and then delete and
