Re: [openstack-dev] request-id in API response

2013-12-11 Thread Sean Dague
On 12/11/2013 04:17 PM, Chris Buccella wrote:
 On 12/02/2013 10:18 AM, Joe Gordon wrote:

  


 Thanks for bringing this up, and I'd welcome a patch in Swift that
 would use a common library to generate the transaction id, if it
 were installed. I can see that there would be huge advantage to
 operators to trace requests through multiple systems.

 Another option would be for each system that calls an another
 OpenStack system to expect and log the transaction ID for the
 request that was given. This would be looser coupling and be more
 forgiving for a heterogeneous cluster. Eg when Glance makes a call
 to Swift, Glance cloud log the transaction id that Swift used
 (from the Swift response). Likewise, when Swift makes a call to
 Keystone, Swift could log the Keystone transaction id. This
 wouldn't result in a single transaction id across all systems, but
 it would provide markers so an admin could trace the request.


 There was a session on this at the summit, and although the notes are
 a little scarce this was the conclusion we came up with.  Every time a
 cross service call is made, we will log and send a notification for
 ceilometer to consume, with the request-ids of both request ids.  One
 of the benefits of this approach is that we can easily generate a tree
 of all the API calls that are made (and clearly show when multiple
 calls are made to the same service), something that just a cross
 service request id would have trouble with.

 https://etherpad.openstack.org/p/icehouse-summit-qa-gate-debugability 


 With that in mind I think having a standard x-openstack-request-id
 makes things a little more uniform, and means that adding new services
 doesn't require new logic to handle new request ids.
 
 Two questions here:
 
 1) The APIChangeGuidelines state that changing a header is frowned upon.
 So I suppose that means we'll need to add x-openstack-request-id to nova
 and cinder, keeping around x-compute-request-id for the time being?
 
 2) The deadline for blueprints for icehouse-2 is next week. This
 blueprint [1] is still marked as next; should be move that up to
 icehouse-2?

x-compute-request-id would need to go through the normal deprecation
path. So deprecate for icehouse, remove in J. Adding
x-openstack-request-id could happen right away, just mirror the ids
across to it.

-Sean

-- 
Sean Dague
http://dague.net

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] request-id in API response

2013-12-06 Thread Joe Gordon
On Dec 6, 2013 9:57 AM, Maru Newby ma...@redhat.com wrote:


 On Dec 6, 2013, at 1:09 AM, John Dickinson m...@not.mn wrote:

 
  On Dec 5, 2013, at 1:36 AM, Maru Newby ma...@redhat.com wrote:
 
 
  On Dec 3, 2013, at 12:18 AM, Joe Gordon joe.gord...@gmail.com wrote:
 
 
 
 
  On Sun, Dec 1, 2013 at 7:04 PM, John Dickinson m...@not.mn wrote:
  Just to add to the story, Swift uses X-Trans-Id and generates it in
the outer-most catch_errors middleware.
 
  Swift's catch errors middleware is responsible for ensuring that the
transaction id exists on each request, and that all errors previously
uncaught, anywhere in the pipeline, are caught and logged. If there is not
a common way to do this, yet, I submit it as a great template for solving
this problem. It's simple, scalable, and well-tested (ie tests and running
in prod for years).
 
 
https://github.com/openstack/swift/blob/master/swift/common/middleware/catch_errors.py
 
  Leaving aside error handling and only focusing on the transaction id
(or request id) generation, since OpenStack services are exposed to
untrusted clients, how would you propose communicating the appropriate
transaction id to a different service? I can see great benefit to having a
glance transaction ID carry through to Swift requests (and so on), but how
should the transaction id be communicated? It's not sensitive info, but I
can imagine a pretty big problem when trying to track down errors if a
client application decides to set eg the X-Set-Transaction-Id header on
every request to the same thing.
 
  -1 to cross service request IDs, for the reasons John mentions above.
 
 
  Thanks for bringing this up, and I'd welcome a patch in Swift that
would use a common library to generate the transaction id, if it were
installed. I can see that there would be huge advantage to operators to
trace requests through multiple systems.
 
  Another option would be for each system that calls an another
OpenStack system to expect and log the transaction ID for the request that
was given. This would be looser coupling and be more forgiving for a
heterogeneous cluster. Eg when Glance makes a call to Swift, Glance cloud
log the transaction id that Swift used (from the Swift response). Likewise,
when Swift makes a call to Keystone, Swift could log the Keystone
transaction id. This wouldn't result in a single transaction id across all
systems, but it would provide markers so an admin could trace the request.
 
  There was a session on this at the summit, and although the notes are
a little scarce this was the conclusion we came up with.  Every time a
cross service call is made, we will log and send a notification for
ceilometer to consume, with the request-ids of both request ids.  One of
the benefits of this approach is that we can easily generate a tree of all
the API calls that are made (and clearly show when multiple calls are made
to the same service), something that just a cross service request id would
have trouble with.
 
  Is wise to trust anything a client provides to ensure traceability?
 If a user receives a request id back from Nova, then submits that request
id in an unrelated request to Neutron, the traceability would be
effectively corrupted.  If the consensus is that we don't want to securely
deliver request ids for inter-service calls, how about requiring a service
to log its request id along with the request id returned from a call to
another service to achieve the a similar result?
 
  Yes, this is what I was proposing. I think this is the best path
forward.

Nova does this for glance client today.  Glance client logs the out glances
request Id and that message is wrapped with novas request id.

Ceilometer wanted notifications of these events as well so it could track
things better.


 Ok, great.  And as per your suggestion, a middleware-based error handler
will soon be proposed for oslo that will secondarily ensure that a request
id is added to the request.

 
 
  The catch is that every call point (or client instantiation?) would
have to be modified to pass the request id instead of just logging at one
place in each service.  Is that a cost worth paying?
 
  Perhaps this is my ignorance of how other projects work today, but does
this not already happen? Is it possible to get a response from an API call
to an OpenStack project that doesn't include a request id?

 We'll have it in Neutron real-soon-now, and then I think the answer will
be 'yes'.

 On reflection, it should be easy enough for a given service to ensure
that calls to other services are automatically instrumented to log request
id pairs.  Again, probably something for oslo.


 m.
 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] request-id in API response

2013-12-06 Thread Akihiro Motoki

(2013/12/06 17:57), Joe Gordon wrote:

 On Dec 6, 2013 9:57 AM, Maru Newby ma...@redhat.com 
 mailto:ma...@redhat.com wrote:
  
  
   On Dec 6, 2013, at 1:09 AM, John Dickinson m...@not.mn 
 mailto:m...@not.mn wrote:
  
   
On Dec 5, 2013, at 1:36 AM, Maru Newby ma...@redhat.com 
 mailto:ma...@redhat.com wrote:
   
   
On Dec 3, 2013, at 12:18 AM, Joe Gordon joe.gord...@gmail.com 
 mailto:joe.gord...@gmail.com wrote:
   
   
   
   
On Sun, Dec 1, 2013 at 7:04 PM, John Dickinson m...@not.mn 
 mailto:m...@not.mn wrote:
Just to add to the story, Swift uses X-Trans-Id and generates it in 
 the outer-most catch_errors middleware.
   
Swift's catch errors middleware is responsible for ensuring that the 
 transaction id exists on each request, and that all errors previously 
 uncaught, anywhere in the pipeline, are caught and logged. If there is not a 
 common way to do this, yet, I submit it as a great template for solving this
 problem. It's simple, scalable, and well-tested (ie tests and running in prod 
 for years).
   

 https://github.com/openstack/swift/blob/master/swift/common/middleware/catch_errors.py
   
Leaving aside error handling and only focusing on the transaction id 
 (or request id) generation, since OpenStack services are exposed to untrusted 
 clients, how would you propose communicating the appropriate transaction id 
 to a different service? I can see great benefit to having a glance
 transaction ID carry through to Swift requests (and so on), but how should 
 the transaction id be communicated? It's not sensitive info, but I can 
 imagine a pretty big problem when trying to track down errors if a client 
 application decides to set eg the X-Set-Transaction-Id header on every request
 to the same thing.
   
-1 to cross service request IDs, for the reasons John mentions above.
   
   
Thanks for bringing this up, and I'd welcome a patch in Swift that 
 would use a common library to generate the transaction id, if it were 
 installed. I can see that there would be huge advantage to operators to trace 
 requests through multiple systems.
   
Another option would be for each system that calls an another 
 OpenStack system to expect and log the transaction ID for the request that 
 was given. This would be looser coupling and be more forgiving for a 
 heterogeneous cluster. Eg when Glance makes a call to Swift, Glance cloud log 
 the
 transaction id that Swift used (from the Swift response). Likewise, when 
 Swift makes a call to Keystone, Swift could log the Keystone transaction id. 
 This wouldn't result in a single transaction id across all systems, but it 
 would provide markers so an admin could trace the request.
   
There was a session on this at the summit, and although the notes are 
 a little scarce this was the conclusion we came up with.  Every time a cross 
 service call is made, we will log and send a notification for ceilometer to 
 consume, with the request-ids of both request ids.  One of the
 benefits of this approach is that we can easily generate a tree of all the 
 API calls that are made (and clearly show when multiple calls are made to the 
 same service), something that just a cross service request id would have 
 trouble with.
   
Is wise to trust anything a client provides to ensure traceability?  If 
 a user receives a request id back from Nova, then submits that request id in 
 an unrelated request to Neutron, the traceability would be effectively 
 corrupted.  If the consensus is that we don't want to securely deliver
 request ids for inter-service calls, how about requiring a service to log its 
 request id along with the request id returned from a call to another service 
 to achieve the a similar result?
   
Yes, this is what I was proposing. I think this is the best path forward.

I think Logging returned request-id at client side is the best way.
We can track each API call even when multiple API calls are invoked in one API 
request.
 
  Nova does this for glance client today.  Glance client logs the out glances 
  request Id and that message is wrapped with novas request id.

When I ran devstack, glance client log (with glance request-id) is not wrapped
with nova request-id: http://paste.openstack.org/show/54590/
It is because glanceclient uses standard logging instead of 
openstack.common.log.
It is common to all client libraries and I am not sure why client libraries
do not use openstack.common.log.

 Ceilometer wanted notifications of these events as well so it could track 
 things better.

Should this feature a part of client libraries or server side which calls 
client libraries?
At now client libraries do not return information about response headers 
including request-id.
If we support it in server side, a standard way to return request-id in a 
response to a caller.
I don't have good idea on it at the moment and it is just a question.

Thanks,
Akihiro


  
   Ok, great.  And as per your suggestion, a 

Re: [openstack-dev] request-id in API response

2013-12-06 Thread Joe Gordon
On Dec 6, 2013 4:26 PM, Akihiro Motoki mot...@da.jp.nec.com wrote:


 (2013/12/06 17:57), Joe Gordon wrote:
 
  On Dec 6, 2013 9:57 AM, Maru Newby ma...@redhat.com mailto:
ma...@redhat.com wrote:
   
   
On Dec 6, 2013, at 1:09 AM, John Dickinson m...@not.mn mailto:
m...@not.mn wrote:
   

 On Dec 5, 2013, at 1:36 AM, Maru Newby ma...@redhat.com mailto:
ma...@redhat.com wrote:


 On Dec 3, 2013, at 12:18 AM, Joe Gordon joe.gord...@gmail.commailto:
joe.gord...@gmail.com wrote:




 On Sun, Dec 1, 2013 at 7:04 PM, John Dickinson m...@not.mnmailto:
m...@not.mn wrote:
 Just to add to the story, Swift uses X-Trans-Id and generates
it in the outer-most catch_errors middleware.

 Swift's catch errors middleware is responsible for ensuring that
the transaction id exists on each request, and that all errors previously
uncaught, anywhere in the pipeline, are caught and logged. If there is not
a common way to do this, yet, I submit it as a great template for solving
this
  problem. It's simple, scalable, and well-tested (ie tests and running
in prod for years).


https://github.com/openstack/swift/blob/master/swift/common/middleware/catch_errors.py

 Leaving aside error handling and only focusing on the
transaction id (or request id) generation, since OpenStack services are
exposed to untrusted clients, how would you propose communicating the
appropriate transaction id to a different service? I can see great benefit
to having a glance
  transaction ID carry through to Swift requests (and so on), but how
should the transaction id be communicated? It's not sensitive info, but I
can imagine a pretty big problem when trying to track down errors if a
client application decides to set eg the X-Set-Transaction-Id header on
every request
  to the same thing.

 -1 to cross service request IDs, for the reasons John mentions
above.


 Thanks for bringing this up, and I'd welcome a patch in Swift
that would use a common library to generate the transaction id, if it were
installed. I can see that there would be huge advantage to operators to
trace requests through multiple systems.

 Another option would be for each system that calls an another
OpenStack system to expect and log the transaction ID for the request that
was given. This would be looser coupling and be more forgiving for a
heterogeneous cluster. Eg when Glance makes a call to Swift, Glance cloud
log the
  transaction id that Swift used (from the Swift response). Likewise,
when Swift makes a call to Keystone, Swift could log the Keystone
transaction id. This wouldn't result in a single transaction id across all
systems, but it would provide markers so an admin could trace the request.

 There was a session on this at the summit, and although the
notes are a little scarce this was the conclusion we came up with.  Every
time a cross service call is made, we will log and send a notification for
ceilometer to consume, with the request-ids of both request ids.  One of the
  benefits of this approach is that we can easily generate a tree of all
the API calls that are made (and clearly show when multiple calls are made
to the same service), something that just a cross service request id would
have trouble with.

 Is wise to trust anything a client provides to ensure
traceability?  If a user receives a request id back from Nova, then submits
that request id in an unrelated request to Neutron, the traceability would
be effectively corrupted.  If the consensus is that we don't want to
securely deliver
  request ids for inter-service calls, how about requiring a service to
log its request id along with the request id returned from a call to
another service to achieve the a similar result?

 Yes, this is what I was proposing. I think this is the best path
forward.

 I think Logging returned request-id at client side is the best way.
 We can track each API call even when multiple API calls are invoked in
one API request.
  
   Nova does this for glance client today.  Glance client logs the out
glances request Id and that message is wrapped with novas request id.

 When I ran devstack, glance client log (with glance request-id) is not
wrapped
 with nova request-id: http://paste.openstack.org/show/54590/
 It is because glanceclient uses standard logging instead of
openstack.common.log.
 It is common to all client libraries and I am not sure why client
libraries
 do not use openstack.common.log.

Woops you are correct, we log both request ids just not on the same line

 
  Ceilometer wanted notifications of these events as well so it could
track things better.

 Should this feature a part of client libraries or server side which calls
client libraries?

Server side

 At now client libraries do not return information about response headers
including request-id.
 If we support it in server side, a standard way to return request-id in a
response to a caller.
 I 

Re: [openstack-dev] request-id in API response

2013-12-05 Thread Maru Newby

On Dec 3, 2013, at 12:18 AM, Joe Gordon joe.gord...@gmail.com wrote:

 
 
 
 On Sun, Dec 1, 2013 at 7:04 PM, John Dickinson m...@not.mn wrote:
 Just to add to the story, Swift uses X-Trans-Id and generates it in the 
 outer-most catch_errors middleware.
 
 Swift's catch errors middleware is responsible for ensuring that the 
 transaction id exists on each request, and that all errors previously 
 uncaught, anywhere in the pipeline, are caught and logged. If there is not a 
 common way to do this, yet, I submit it as a great template for solving this 
 problem. It's simple, scalable, and well-tested (ie tests and running in prod 
 for years).
 
 https://github.com/openstack/swift/blob/master/swift/common/middleware/catch_errors.py
 
 Leaving aside error handling and only focusing on the transaction id (or 
 request id) generation, since OpenStack services are exposed to untrusted 
 clients, how would you propose communicating the appropriate transaction id 
 to a different service? I can see great benefit to having a glance 
 transaction ID carry through to Swift requests (and so on), but how should 
 the transaction id be communicated? It's not sensitive info, but I can 
 imagine a pretty big problem when trying to track down errors if a client 
 application decides to set eg the X-Set-Transaction-Id header on every 
 request to the same thing.
 
 -1 to cross service request IDs, for the reasons John mentions above.
 
 
 Thanks for bringing this up, and I'd welcome a patch in Swift that would use 
 a common library to generate the transaction id, if it were installed. I can 
 see that there would be huge advantage to operators to trace requests through 
 multiple systems.
 
 Another option would be for each system that calls an another OpenStack 
 system to expect and log the transaction ID for the request that was given. 
 This would be looser coupling and be more forgiving for a heterogeneous 
 cluster. Eg when Glance makes a call to Swift, Glance cloud log the 
 transaction id that Swift used (from the Swift response). Likewise, when 
 Swift makes a call to Keystone, Swift could log the Keystone transaction id. 
 This wouldn't result in a single transaction id across all systems, but it 
 would provide markers so an admin could trace the request.
 
 There was a session on this at the summit, and although the notes are a 
 little scarce this was the conclusion we came up with.  Every time a cross 
 service call is made, we will log and send a notification for ceilometer to 
 consume, with the request-ids of both request ids.  One of the benefits of 
 this approach is that we can easily generate a tree of all the API calls that 
 are made (and clearly show when multiple calls are made to the same service), 
 something that just a cross service request id would have trouble with.

Is wise to trust anything a client provides to ensure traceability?  If a user 
receives a request id back from Nova, then submits that request id in an 
unrelated request to Neutron, the traceability would be effectively corrupted.  
If the consensus is that we don't want to securely deliver request ids for 
inter-service calls, how about requiring a service to log its request id along 
with the request id returned from a call to another service to achieve the a 
similar result?  The catch is that every call point (or client instantiation?) 
would have to be modified to pass the request id instead of just logging at one 
place in each service.  Is that a cost worth paying?


m.


 
 https://etherpad.openstack.org/p/icehouse-summit-qa-gate-debugability 
 
 
 With that in mind I think having a standard x-openstack-request-id makes 
 things a little more uniform, and means that adding new services doesn't 
 require new logic to handle new request ids.





___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] request-id in API response

2013-12-05 Thread John Dickinson

On Dec 5, 2013, at 1:36 AM, Maru Newby ma...@redhat.com wrote:

 
 On Dec 3, 2013, at 12:18 AM, Joe Gordon joe.gord...@gmail.com wrote:
 
 
 
 
 On Sun, Dec 1, 2013 at 7:04 PM, John Dickinson m...@not.mn wrote:
 Just to add to the story, Swift uses X-Trans-Id and generates it in the 
 outer-most catch_errors middleware.
 
 Swift's catch errors middleware is responsible for ensuring that the 
 transaction id exists on each request, and that all errors previously 
 uncaught, anywhere in the pipeline, are caught and logged. If there is not a 
 common way to do this, yet, I submit it as a great template for solving this 
 problem. It's simple, scalable, and well-tested (ie tests and running in 
 prod for years).
 
 https://github.com/openstack/swift/blob/master/swift/common/middleware/catch_errors.py
 
 Leaving aside error handling and only focusing on the transaction id (or 
 request id) generation, since OpenStack services are exposed to untrusted 
 clients, how would you propose communicating the appropriate transaction id 
 to a different service? I can see great benefit to having a glance 
 transaction ID carry through to Swift requests (and so on), but how should 
 the transaction id be communicated? It's not sensitive info, but I can 
 imagine a pretty big problem when trying to track down errors if a client 
 application decides to set eg the X-Set-Transaction-Id header on every 
 request to the same thing.
 
 -1 to cross service request IDs, for the reasons John mentions above.
 
 
 Thanks for bringing this up, and I'd welcome a patch in Swift that would use 
 a common library to generate the transaction id, if it were installed. I can 
 see that there would be huge advantage to operators to trace requests 
 through multiple systems.
 
 Another option would be for each system that calls an another OpenStack 
 system to expect and log the transaction ID for the request that was given. 
 This would be looser coupling and be more forgiving for a heterogeneous 
 cluster. Eg when Glance makes a call to Swift, Glance cloud log the 
 transaction id that Swift used (from the Swift response). Likewise, when 
 Swift makes a call to Keystone, Swift could log the Keystone transaction id. 
 This wouldn't result in a single transaction id across all systems, but it 
 would provide markers so an admin could trace the request.
 
 There was a session on this at the summit, and although the notes are a 
 little scarce this was the conclusion we came up with.  Every time a cross 
 service call is made, we will log and send a notification for ceilometer to 
 consume, with the request-ids of both request ids.  One of the benefits of 
 this approach is that we can easily generate a tree of all the API calls 
 that are made (and clearly show when multiple calls are made to the same 
 service), something that just a cross service request id would have trouble 
 with.
 
 Is wise to trust anything a client provides to ensure traceability?  If a 
 user receives a request id back from Nova, then submits that request id in an 
 unrelated request to Neutron, the traceability would be effectively 
 corrupted.  If the consensus is that we don't want to securely deliver 
 request ids for inter-service calls, how about requiring a service to log its 
 request id along with the request id returned from a call to another service 
 to achieve the a similar result?

Yes, this is what I was proposing. I think this is the best path forward.


 The catch is that every call point (or client instantiation?) would have to 
 be modified to pass the request id instead of just logging at one place in 
 each service.  Is that a cost worth paying?

Perhaps this is my ignorance of how other projects work today, but does this 
not already happen? Is it possible to get a response from an API call to an 
OpenStack project that doesn't include a request id?

 
 
 m.
 
 
 
 https://etherpad.openstack.org/p/icehouse-summit-qa-gate-debugability 
 
 
 With that in mind I think having a standard x-openstack-request-id makes 
 things a little more uniform, and means that adding new services doesn't 
 require new logic to handle new request ids.
 
 
 
 
 
 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] request-id in API response

2013-12-05 Thread Maru Newby

On Dec 6, 2013, at 1:09 AM, John Dickinson m...@not.mn wrote:

 
 On Dec 5, 2013, at 1:36 AM, Maru Newby ma...@redhat.com wrote:
 
 
 On Dec 3, 2013, at 12:18 AM, Joe Gordon joe.gord...@gmail.com wrote:
 
 
 
 
 On Sun, Dec 1, 2013 at 7:04 PM, John Dickinson m...@not.mn wrote:
 Just to add to the story, Swift uses X-Trans-Id and generates it in the 
 outer-most catch_errors middleware.
 
 Swift's catch errors middleware is responsible for ensuring that the 
 transaction id exists on each request, and that all errors previously 
 uncaught, anywhere in the pipeline, are caught and logged. If there is not 
 a common way to do this, yet, I submit it as a great template for solving 
 this problem. It's simple, scalable, and well-tested (ie tests and running 
 in prod for years).
 
 https://github.com/openstack/swift/blob/master/swift/common/middleware/catch_errors.py
 
 Leaving aside error handling and only focusing on the transaction id (or 
 request id) generation, since OpenStack services are exposed to untrusted 
 clients, how would you propose communicating the appropriate transaction id 
 to a different service? I can see great benefit to having a glance 
 transaction ID carry through to Swift requests (and so on), but how should 
 the transaction id be communicated? It's not sensitive info, but I can 
 imagine a pretty big problem when trying to track down errors if a client 
 application decides to set eg the X-Set-Transaction-Id header on every 
 request to the same thing.
 
 -1 to cross service request IDs, for the reasons John mentions above.
 
 
 Thanks for bringing this up, and I'd welcome a patch in Swift that would 
 use a common library to generate the transaction id, if it were installed. 
 I can see that there would be huge advantage to operators to trace requests 
 through multiple systems.
 
 Another option would be for each system that calls an another OpenStack 
 system to expect and log the transaction ID for the request that was given. 
 This would be looser coupling and be more forgiving for a heterogeneous 
 cluster. Eg when Glance makes a call to Swift, Glance cloud log the 
 transaction id that Swift used (from the Swift response). Likewise, when 
 Swift makes a call to Keystone, Swift could log the Keystone transaction 
 id. This wouldn't result in a single transaction id across all systems, but 
 it would provide markers so an admin could trace the request.
 
 There was a session on this at the summit, and although the notes are a 
 little scarce this was the conclusion we came up with.  Every time a cross 
 service call is made, we will log and send a notification for ceilometer to 
 consume, with the request-ids of both request ids.  One of the benefits of 
 this approach is that we can easily generate a tree of all the API calls 
 that are made (and clearly show when multiple calls are made to the same 
 service), something that just a cross service request id would have trouble 
 with.
 
 Is wise to trust anything a client provides to ensure traceability?  If a 
 user receives a request id back from Nova, then submits that request id in 
 an unrelated request to Neutron, the traceability would be effectively 
 corrupted.  If the consensus is that we don't want to securely deliver 
 request ids for inter-service calls, how about requiring a service to log 
 its request id along with the request id returned from a call to another 
 service to achieve the a similar result?
 
 Yes, this is what I was proposing. I think this is the best path forward.

Ok, great.  And as per your suggestion, a middleware-based error handler will 
soon be proposed for oslo that will secondarily ensure that a request id is 
added to the request.  

 
 
 The catch is that every call point (or client instantiation?) would have to 
 be modified to pass the request id instead of just logging at one place in 
 each service.  Is that a cost worth paying?
 
 Perhaps this is my ignorance of how other projects work today, but does this 
 not already happen? Is it possible to get a response from an API call to an 
 OpenStack project that doesn't include a request id?

We'll have it in Neutron real-soon-now, and then I think the answer will be 
'yes'.

On reflection, it should be easy enough for a given service to ensure that 
calls to other services are automatically instrumented to log request id pairs. 
 Again, probably something for oslo.


m.
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] request-id in API response

2013-12-02 Thread Jay Pipes

On 12/01/2013 10:04 PM, John Dickinson wrote:

Just to add to the story, Swift uses X-Trans-Id and generates it in
the outer-most catch_errors middleware.

Swift's catch errors middleware is responsible for ensuring that the
transaction id exists on each request, and that all errors previously
uncaught, anywhere in the pipeline, are caught and logged. If there
is not a common way to do this, yet, I submit it as a great template
for solving this problem. It's simple, scalable, and well-tested (ie
tests and running in prod for years).

https://github.com/openstack/swift/blob/master/swift/common/middleware/catch_errors.py


++

If there's prior art here, might as well use it. I'm not a huge fan of 
using the term transaction within things that do not have a 
transactional safety context... but that's just because of my background 
in RDBMS stuff. If X-Trans-Id is already in use by another OpenStack 
project, it should probably take precedence over something new unless 
there is a really good reason otherwise (and my personal opinion about 
the semantics of transactions ain't a good reason!).



 Leaving aside error handling and only focusing on the transaction id
(or request id) generation, since OpenStack services are exposed to
untrusted clients, how would you propose communicating the
appropriate transaction id to a different service? I can see great
benefit to having a glance transaction ID carry through to Swift
requests (and so on), but how should the transaction id be
communicated? It's not sensitive info, but I can imagine a pretty big
problem when trying to track down errors if a client application
decides to set eg the X-Set-Transaction-Id header on every request to
the same thing.


I suppose if this were really a problem (and I'm not sold on the idea 
that it is a problem), one solution might be to store a checksum 
somewhere for the transaction ID and some other piece of data. But I 
don't really see that as super useful, and it would slow things down. 
Glance already stores a checksum for important things like the data in 
an image. If a service needs to be absolutely sure that a piece of data 
hasn't been messed with, this cross-service request ID probably isn't 
the thing to use...



Thanks for bringing this up, and I'd welcome a patch in Swift that
would use a common library to generate the transaction id, if it were
installed. I can see that there would be huge advantage to operators
to trace requests through multiple systems.


Hmm, so does that mean that you'd be open to (gradually) moving to an 
x-openstack-request-id header to replace x-trans-id?



Another option would be for each system that calls an another
OpenStack system to expect and log the transaction ID for the request
that was given. This would be looser coupling and be more forgiving
for a heterogeneous cluster. Eg when Glance makes a call to Swift,
Glance cloud log the transaction id that Swift used (from the Swift
response). Likewise, when Swift makes a call to Keystone, Swift could
log the Keystone transaction id. This wouldn't result in a single
transaction id across all systems, but it would provide markers so an
admin could trace the request.


Sure, this is a perfectly fine option, but doesn't really provide the 
single traceable ID value that I think we're looking for here.


Best,
-jay


On Dec 1, 2013, at 5:48 PM, Maru Newby ma...@redhat.com wrote:



On Nov 30, 2013, at 1:00 AM, Sean Dague s...@dague.net wrote:


On 11/29/2013 10:33 AM, Jay Pipes wrote:

On 11/28/2013 07:45 AM, Akihiro Motoki wrote:

Hi,

I am working on adding request-id to API response in
Neutron. After I checked what header is used in other
projects header name varies project by project. It seems
there is no consensus what header is recommended and it is
better to have some consensus.

nova: x-compute-request-id cinder:
x-compute-request-id glance:   x-openstack-request-id
neutron:  x-network-request-id  (under review)

request-id is assigned and used inside of each project now,
so x-service-request-id looks good. On the other hand, if
we have a plan to enhance request-id across projects,
x-openstack-request-id looks better.


My vote is for:

x-openstack-request-id

With an implementation of create a request UUID if none exists
yet in some standardized WSGI middleware...


Agreed. I don't think I see any value in having these have
different service names, having just x-openstack-request-id
across all the services seems a far better idea, and come back
through and fix nova and cinder to be that as well.


+1

An openstack request id should be service agnostic to allow
tracking of a request across many services (e.g. a call to nova to
boot a VM should generate a request id that is provided to other
services in requests to provision said VM).  All services would
ideally share a facility for generating new request ids and for
securely accepting request ids from other services.


m.



-Sean

-- Sean Dague http://dague.net


Re: [openstack-dev] request-id in API response

2013-12-02 Thread Joe Gordon
On Sun, Dec 1, 2013 at 7:04 PM, John Dickinson m...@not.mn wrote:

 Just to add to the story, Swift uses X-Trans-Id and generates it in the
 outer-most catch_errors middleware.

 Swift's catch errors middleware is responsible for ensuring that the
 transaction id exists on each request, and that all errors previously
 uncaught, anywhere in the pipeline, are caught and logged. If there is not
 a common way to do this, yet, I submit it as a great template for solving
 this problem. It's simple, scalable, and well-tested (ie tests and running
 in prod for years).


 https://github.com/openstack/swift/blob/master/swift/common/middleware/catch_errors.py

 Leaving aside error handling and only focusing on the transaction id (or
 request id) generation, since OpenStack services are exposed to untrusted
 clients, how would you propose communicating the appropriate transaction id
 to a different service? I can see great benefit to having a glance
 transaction ID carry through to Swift requests (and so on), but how should
 the transaction id be communicated? It's not sensitive info, but I can
 imagine a pretty big problem when trying to track down errors if a client
 application decides to set eg the X-Set-Transaction-Id header on every
 request to the same thing.


-1 to cross service request IDs, for the reasons John mentions above.



 Thanks for bringing this up, and I'd welcome a patch in Swift that would
 use a common library to generate the transaction id, if it were installed.
 I can see that there would be huge advantage to operators to trace requests
 through multiple systems.

 Another option would be for each system that calls an another OpenStack
 system to expect and log the transaction ID for the request that was given.
 This would be looser coupling and be more forgiving for a heterogeneous
 cluster. Eg when Glance makes a call to Swift, Glance cloud log the
 transaction id that Swift used (from the Swift response). Likewise, when
 Swift makes a call to Keystone, Swift could log the Keystone transaction
 id. This wouldn't result in a single transaction id across all systems, but
 it would provide markers so an admin could trace the request.


There was a session on this at the summit, and although the notes are a
little scarce this was the conclusion we came up with.  Every time a cross
service call is made, we will log and send a notification for ceilometer to
consume, with the request-ids of both request ids.  One of the benefits of
this approach is that we can easily generate a tree of all the API calls
that are made (and clearly show when multiple calls are made to the same
service), something that just a cross service request id would have trouble
with.

https://etherpad.openstack.org/p/icehouse-summit-qa-gate-debugability


With that in mind I think having a standard x-openstack-request-id makes
things a little more uniform, and means that adding new services doesn't
require new logic to handle new request ids.



 --John




 On Dec 1, 2013, at 5:48 PM, Maru Newby ma...@redhat.com wrote:

 
  On Nov 30, 2013, at 1:00 AM, Sean Dague s...@dague.net wrote:
 
  On 11/29/2013 10:33 AM, Jay Pipes wrote:
  On 11/28/2013 07:45 AM, Akihiro Motoki wrote:
  Hi,
 
  I am working on adding request-id to API response in Neutron.
  After I checked what header is used in other projects
  header name varies project by project.
  It seems there is no consensus what header is recommended
  and it is better to have some consensus.
 
  nova: x-compute-request-id
  cinder:   x-compute-request-id
  glance:   x-openstack-request-id
  neutron:  x-network-request-id  (under review)
 
  request-id is assigned and used inside of each project now,
  so x-service-request-id looks good. On the other hand,
  if we have a plan to enhance request-id across projects,
  x-openstack-request-id looks better.
 
  My vote is for:
 
  x-openstack-request-id
 
  With an implementation of create a request UUID if none exists yet in
  some standardized WSGI middleware...
 
  Agreed. I don't think I see any value in having these have different
  service names, having just x-openstack-request-id across all the
  services seems a far better idea, and come back through and fix nova and
  cinder to be that as well.
 
  +1
 
  An openstack request id should be service agnostic to allow tracking of
 a request across many services (e.g. a call to nova to boot a VM should
 generate a request id that is provided to other services in requests to
 provision said VM).  All services would ideally share a facility for
 generating new request ids and for securely accepting request ids from
 other services.
 
 
  m.
 
 
   -Sean
 
  --
  Sean Dague
  http://dague.net
 
  ___
  OpenStack-dev mailing list
  OpenStack-dev@lists.openstack.org
  http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
 
 
  ___
  OpenStack-dev mailing list
  

Re: [openstack-dev] request-id in API response

2013-12-01 Thread Maru Newby

On Nov 30, 2013, at 1:00 AM, Sean Dague s...@dague.net wrote:

 On 11/29/2013 10:33 AM, Jay Pipes wrote:
 On 11/28/2013 07:45 AM, Akihiro Motoki wrote:
 Hi,
 
 I am working on adding request-id to API response in Neutron.
 After I checked what header is used in other projects
 header name varies project by project.
 It seems there is no consensus what header is recommended
 and it is better to have some consensus.
 
   nova: x-compute-request-id
   cinder:   x-compute-request-id
   glance:   x-openstack-request-id
   neutron:  x-network-request-id  (under review)
 
 request-id is assigned and used inside of each project now,
 so x-service-request-id looks good. On the other hand,
 if we have a plan to enhance request-id across projects,
 x-openstack-request-id looks better.
 
 My vote is for:
 
 x-openstack-request-id
 
 With an implementation of create a request UUID if none exists yet in
 some standardized WSGI middleware...
 
 Agreed. I don't think I see any value in having these have different
 service names, having just x-openstack-request-id across all the
 services seems a far better idea, and come back through and fix nova and
 cinder to be that as well.

+1 

An openstack request id should be service agnostic to allow tracking of a 
request across many services (e.g. a call to nova to boot a VM should generate 
a request id that is provided to other services in requests to provision said 
VM).  All services would ideally share a facility for generating new request 
ids and for securely accepting request ids from other services.


m.

 
   -Sean
 
 -- 
 Sean Dague
 http://dague.net
 
 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] request-id in API response

2013-12-01 Thread John Dickinson
Just to add to the story, Swift uses X-Trans-Id and generates it in the 
outer-most catch_errors middleware.

Swift's catch errors middleware is responsible for ensuring that the 
transaction id exists on each request, and that all errors previously uncaught, 
anywhere in the pipeline, are caught and logged. If there is not a common way 
to do this, yet, I submit it as a great template for solving this problem. It's 
simple, scalable, and well-tested (ie tests and running in prod for years).

https://github.com/openstack/swift/blob/master/swift/common/middleware/catch_errors.py

Leaving aside error handling and only focusing on the transaction id (or 
request id) generation, since OpenStack services are exposed to untrusted 
clients, how would you propose communicating the appropriate transaction id to 
a different service? I can see great benefit to having a glance transaction ID 
carry through to Swift requests (and so on), but how should the transaction id 
be communicated? It's not sensitive info, but I can imagine a pretty big 
problem when trying to track down errors if a client application decides to set 
eg the X-Set-Transaction-Id header on every request to the same thing.

Thanks for bringing this up, and I'd welcome a patch in Swift that would use a 
common library to generate the transaction id, if it were installed. I can see 
that there would be huge advantage to operators to trace requests through 
multiple systems.

Another option would be for each system that calls an another OpenStack system 
to expect and log the transaction ID for the request that was given. This would 
be looser coupling and be more forgiving for a heterogeneous cluster. Eg when 
Glance makes a call to Swift, Glance cloud log the transaction id that Swift 
used (from the Swift response). Likewise, when Swift makes a call to Keystone, 
Swift could log the Keystone transaction id. This wouldn't result in a single 
transaction id across all systems, but it would provide markers so an admin 
could trace the request.

--John




On Dec 1, 2013, at 5:48 PM, Maru Newby ma...@redhat.com wrote:

 
 On Nov 30, 2013, at 1:00 AM, Sean Dague s...@dague.net wrote:
 
 On 11/29/2013 10:33 AM, Jay Pipes wrote:
 On 11/28/2013 07:45 AM, Akihiro Motoki wrote:
 Hi,
 
 I am working on adding request-id to API response in Neutron.
 After I checked what header is used in other projects
 header name varies project by project.
 It seems there is no consensus what header is recommended
 and it is better to have some consensus.
 
 nova: x-compute-request-id
 cinder:   x-compute-request-id
 glance:   x-openstack-request-id
 neutron:  x-network-request-id  (under review)
 
 request-id is assigned and used inside of each project now,
 so x-service-request-id looks good. On the other hand,
 if we have a plan to enhance request-id across projects,
 x-openstack-request-id looks better.
 
 My vote is for:
 
 x-openstack-request-id
 
 With an implementation of create a request UUID if none exists yet in
 some standardized WSGI middleware...
 
 Agreed. I don't think I see any value in having these have different
 service names, having just x-openstack-request-id across all the
 services seems a far better idea, and come back through and fix nova and
 cinder to be that as well.
 
 +1 
 
 An openstack request id should be service agnostic to allow tracking of a 
 request across many services (e.g. a call to nova to boot a VM should 
 generate a request id that is provided to other services in requests to 
 provision said VM).  All services would ideally share a facility for 
 generating new request ids and for securely accepting request ids from other 
 services.
 
 
 m.
 
 
  -Sean
 
 -- 
 Sean Dague
 http://dague.net
 
 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
 
 
 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] request-id in API response

2013-11-29 Thread Jay Pipes

On 11/28/2013 07:45 AM, Akihiro Motoki wrote:

Hi,

I am working on adding request-id to API response in Neutron.
After I checked what header is used in other projects
header name varies project by project.
It seems there is no consensus what header is recommended
and it is better to have some consensus.

   nova: x-compute-request-id
   cinder:   x-compute-request-id
   glance:   x-openstack-request-id
   neutron:  x-network-request-id  (under review)

request-id is assigned and used inside of each project now,
so x-service-request-id looks good. On the other hand,
if we have a plan to enhance request-id across projects,
x-openstack-request-id looks better.


My vote is for:

x-openstack-request-id

With an implementation of create a request UUID if none exists yet in 
some standardized WSGI middleware...


Best,
-jay

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] request-id in API response

2013-11-29 Thread Sean Dague
On 11/29/2013 10:33 AM, Jay Pipes wrote:
 On 11/28/2013 07:45 AM, Akihiro Motoki wrote:
 Hi,

 I am working on adding request-id to API response in Neutron.
 After I checked what header is used in other projects
 header name varies project by project.
 It seems there is no consensus what header is recommended
 and it is better to have some consensus.

nova: x-compute-request-id
cinder:   x-compute-request-id
glance:   x-openstack-request-id
neutron:  x-network-request-id  (under review)

 request-id is assigned and used inside of each project now,
 so x-service-request-id looks good. On the other hand,
 if we have a plan to enhance request-id across projects,
 x-openstack-request-id looks better.
 
 My vote is for:
 
 x-openstack-request-id
 
 With an implementation of create a request UUID if none exists yet in
 some standardized WSGI middleware...

Agreed. I don't think I see any value in having these have different
service names, having just x-openstack-request-id across all the
services seems a far better idea, and come back through and fix nova and
cinder to be that as well.

-Sean

-- 
Sean Dague
http://dague.net



signature.asc
Description: OpenPGP digital signature
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev