Re: [openstack-dev] [Magnum][Heat] Expression of Bay Status
In summary, we have decided to use the usage notification events emitted by heat (http://tinyurl.com/oultjm5). This should allow us to detect stack action completions, and errors (with reasons) which should be enough for a Bay implementation that does not need to poll. Stack timeouts can be passed to heat as parameters. This is not as elegant as what Angus and Zane suggested because it requires Magnum to access all RPC messages. With their proposed approach we could use a Zaqar queue that only emits the stack events relevant to that user/tenant. This conforms better to the principle of least privilege. Our preference for the decided approach is that it allows us tight integration with Heat using functionality that exists today. We admit that this implementation will be harder to debug than other options. More remarks in-line. On Mar 11, 2015, at 2:27 AM, Jay Lau jay.lau@gmail.com wrote: 2015-03-10 23:21 GMT+08:00 Hongbin Lu hongbin...@gmail.com: Hi Adrian, On Mon, Mar 9, 2015 at 6:53 PM, Adrian Otto adrian.o...@rackspace.com wrote: Magnum Team, In the following review, we have the start of a discussion about how to tackle bay status: https://review.openstack.org/159546 I think a key issue here is that we are not subscribing to an event feed from Heat to tell us about each state transition, so we have a low degree of confidence that our state will match the actual state of the stack in real-time. At best, we have an eventually consistent state for Bay following a bay creation. Here are some options for us to consider to solve this: 1) Propose enhancements to Heat (or learn about existing features) to emit a set of notifications upon state changes to stack resources so the state can be mirrored in the Bay resource. A drawback of this option is that it increases the difficulty of trouble-shooting. In my experience of using Heat (SoftwareDeployments in particular), Ironic and Trove, one of the most frequent errors I encountered is that the provisioning resources stayed in deploying state (never went to completed). The reason is that they were waiting a callback signal from the provisioning resource to indicate its completion, but the callback signal was blocked due to various reasons (e.g. incorrect firewall rules, incorrect configs, etc.). Troubling-shooting such problem is generally harder. I think that the heat convergence is working on the issues for your concern: https://wiki.openstack.org/wiki/Heat/Blueprints/Convergence Yes, that wold address part of the concern, but not all of it. Depending on the implementation of state exchange, and the configuration of the network, it may be possible to create an installation of the system that does not function at all due to network ACLs, and almost no clue to the implementer about why it does not work. To mitigate this concern, we would want an implementation that does not require a bi-directional network trust relationship between Heat and Magnum. Instead, implement it in a way that connections are only made from Magnum to Heat, so if the stack creation call succeeds, so can the status updates. Perhaps we can revisit this design discussion in a future iteration of our Bay status code. Adrian 2) Spawn a task to poll the Heat stack resource for state changes, and express them in the Bay status, and allow that task to exit once the stack reaches its terminal (completed) state. 3) Don’t store any state in the Bay object, and simply query the heat stack for status as needed. Are each of these options viable? Are there other options to consider? What are the pro/con arguments for each? Thanks, Adrian __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Thanks, Jay Lau (Guangya Liu) __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Magnum][Heat] Expression of Bay Status
2015-03-10 23:21 GMT+08:00 Hongbin Lu hongbin...@gmail.com: Hi Adrian, On Mon, Mar 9, 2015 at 6:53 PM, Adrian Otto adrian.o...@rackspace.com wrote: Magnum Team, In the following review, we have the start of a discussion about how to tackle bay status: https://review.openstack.org/159546 I think a key issue here is that we are not subscribing to an event feed from Heat to tell us about each state transition, so we have a low degree of confidence that our state will match the actual state of the stack in real-time. At best, we have an eventually consistent state for Bay following a bay creation. Here are some options for us to consider to solve this: 1) Propose enhancements to Heat (or learn about existing features) to emit a set of notifications upon state changes to stack resources so the state can be mirrored in the Bay resource. A drawback of this option is that it increases the difficulty of trouble-shooting. In my experience of using Heat (SoftwareDeployments in particular), Ironic and Trove, one of the most frequent errors I encountered is that the provisioning resources stayed in deploying state (never went to completed). The reason is that they were waiting a callback signal from the provisioning resource to indicate its completion, but the callback signal was blocked due to various reasons (e.g. incorrect firewall rules, incorrect configs, etc.). Troubling-shooting such problem is generally harder. I think that the heat convergence is working on the issues for your concern: https://wiki.openstack.org/wiki/Heat/Blueprints/Convergence 2) Spawn a task to poll the Heat stack resource for state changes, and express them in the Bay status, and allow that task to exit once the stack reaches its terminal (completed) state. 3) Don’t store any state in the Bay object, and simply query the heat stack for status as needed. Are each of these options viable? Are there other options to consider? What are the pro/con arguments for each? Thanks, Adrian __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Thanks, Jay Lau (Guangya Liu) __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Magnum][Heat] Expression of Bay Status
On 09/03/15 23:47, Angus Salkeld wrote: On Tue, Mar 10, 2015 at 8:53 AM, Adrian Otto adrian.o...@rackspace.com mailto:adrian.o...@rackspace.com wrote: Magnum Team, In the following review, we have the start of a discussion about how to tackle bay status: https://review.openstack.org/159546 I think a key issue here is that we are not subscribing to an event feed from Heat to tell us about each state transition, so we have a low degree of confidence that our state will match the actual state of the stack in real-time. At best, we have an eventually consistent state for Bay following a bay creation. Hi Adrian Currently Heat does not an event stream, but instead an event table and REST resource. This sucks as you have to poll it. We have been long wanting some integration with Zaqar - we are all convinced, just needs someone to do the work. So the idea here is we send user related events via a Zaqar queue and the user (Magnum) subscribes and get events. From last summit https://etherpad.openstack.org/p/kilo-heat-summit-topics (see line 73). +1 Here are some options for us to consider to solve this: 1) Propose enhancements to Heat (or learn about existing features) to emit a set of notifications upon state changes to stack resources so the state can be mirrored in the Bay resource. See above, you have anyone to drive this? 2) Spawn a task to poll the Heat stack resource for state changes, and express them in the Bay status, and allow that task to exit once the stack reaches its terminal (completed) state. 3) Don’t store any state in the Bay object, and simply query the heat stack for status as needed. If it's not too frequent then this might be your best bet until we get 1). Hope this helps -Angus Are each of these options viable? Are there other options to consider? What are the pro/con arguments for each? Thanks, Adrian __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Magnum][Heat] Expression of Bay Status
Hi Adrian, On Mon, Mar 9, 2015 at 6:53 PM, Adrian Otto adrian.o...@rackspace.com wrote: Magnum Team, In the following review, we have the start of a discussion about how to tackle bay status: https://review.openstack.org/159546 I think a key issue here is that we are not subscribing to an event feed from Heat to tell us about each state transition, so we have a low degree of confidence that our state will match the actual state of the stack in real-time. At best, we have an eventually consistent state for Bay following a bay creation. Here are some options for us to consider to solve this: 1) Propose enhancements to Heat (or learn about existing features) to emit a set of notifications upon state changes to stack resources so the state can be mirrored in the Bay resource. A drawback of this option is that it increases the difficulty of trouble-shooting. In my experience of using Heat (SoftwareDeployments in particular), Ironic and Trove, one of the most frequent errors I encountered is that the provisioning resources stayed in deploying state (never went to completed). The reason is that they were waiting a callback signal from the provisioning resource to indicate its completion, but the callback signal was blocked due to various reasons (e.g. incorrect firewall rules, incorrect configs, etc.). Troubling-shooting such problem is generally harder. 2) Spawn a task to poll the Heat stack resource for state changes, and express them in the Bay status, and allow that task to exit once the stack reaches its terminal (completed) state. 3) Don’t store any state in the Bay object, and simply query the heat stack for status as needed. Are each of these options viable? Are there other options to consider? What are the pro/con arguments for each? Thanks, Adrian __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [Magnum][Heat] Expression of Bay Status
Magnum Team, In the following review, we have the start of a discussion about how to tackle bay status: https://review.openstack.org/159546 I think a key issue here is that we are not subscribing to an event feed from Heat to tell us about each state transition, so we have a low degree of confidence that our state will match the actual state of the stack in real-time. At best, we have an eventually consistent state for Bay following a bay creation. Here are some options for us to consider to solve this: 1) Propose enhancements to Heat (or learn about existing features) to emit a set of notifications upon state changes to stack resources so the state can be mirrored in the Bay resource. 2) Spawn a task to poll the Heat stack resource for state changes, and express them in the Bay status, and allow that task to exit once the stack reaches its terminal (completed) state. 3) Don’t store any state in the Bay object, and simply query the heat stack for status as needed. Are each of these options viable? Are there other options to consider? What are the pro/con arguments for each? Thanks, Adrian __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Magnum][Heat] Expression of Bay Status
Hi Adrian. Anyway, I think `Taskflow` is useful to manage jobs which take a long time. So if we don’t use taskflow to manage bay status, we should use this to manage k8s resources. -- OTSUKA, Motohiro Sent with Sparrow (http://www.sparrowmailapp.com/?sig) On Tuesday, March 10, 2015 at 07:53, Adrian Otto wrote: Magnum Team, In the following review, we have the start of a discussion about how to tackle bay status: https://review.openstack.org/159546 I think a key issue here is that we are not subscribing to an event feed from Heat to tell us about each state transition, so we have a low degree of confidence that our state will match the actual state of the stack in real-time. At best, we have an eventually consistent state for Bay following a bay creation. Here are some options for us to consider to solve this: 1) Propose enhancements to Heat (or learn about existing features) to emit a set of notifications upon state changes to stack resources so the state can be mirrored in the Bay resource. 2) Spawn a task to poll the Heat stack resource for state changes, and express them in the Bay status, and allow that task to exit once the stack reaches its terminal (completed) state. 3) Don’t store any state in the Bay object, and simply query the heat stack for status as needed. Are each of these options viable? Are there other options to consider? What are the pro/con arguments for each? Thanks, Adrian __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe (mailto:openstack-dev-requ...@lists.openstack.org?subject:unsubscribe) http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Magnum][Heat] Expression of Bay Status
On Tue, Mar 10, 2015 at 8:53 AM, Adrian Otto adrian.o...@rackspace.com wrote: Magnum Team, In the following review, we have the start of a discussion about how to tackle bay status: https://review.openstack.org/159546 I think a key issue here is that we are not subscribing to an event feed from Heat to tell us about each state transition, so we have a low degree of confidence that our state will match the actual state of the stack in real-time. At best, we have an eventually consistent state for Bay following a bay creation. Hi Adrian Currently Heat does not an event stream, but instead an event table and REST resource. This sucks as you have to poll it. We have been long wanting some integration with Zaqar - we are all convinced, just needs someone to do the work. So the idea here is we send user related events via a Zaqar queue and the user (Magnum) subscribes and get events. From last summit https://etherpad.openstack.org/p/kilo-heat-summit-topics (see line 73). Here are some options for us to consider to solve this: 1) Propose enhancements to Heat (or learn about existing features) to emit a set of notifications upon state changes to stack resources so the state can be mirrored in the Bay resource. See above, you have anyone to drive this? 2) Spawn a task to poll the Heat stack resource for state changes, and express them in the Bay status, and allow that task to exit once the stack reaches its terminal (completed) state. 3) Don’t store any state in the Bay object, and simply query the heat stack for status as needed. If it's not too frequent then this might be your best bet until we get 1). Hope this helps -Angus Are each of these options viable? Are there other options to consider? What are the pro/con arguments for each? Thanks, Adrian __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
