Re: [openstack-dev] [Heat] Continue discussing multi-region orchestration
Excerpts from Mike Spreitzer's message of 2013-11-20 11:09:34 -0800: > OTOH, the more we restrict what can be done, the less useful this really > is. I would be more specific and say it as "...the less divergent behavior is actually possible." It will be quite a bit more useful if it is boring and restricted but does what it claims to do extremely well. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Heat] Continue discussing multi-region orchestration
On 20/11/13 20:09, Mike Spreitzer wrote: Zane Bitter wrote on 11/15/2013 05:59:06 PM: > On 15/11/13 22:17, Keith Bray wrote: > > The way I view 2 vs. 4 is that 2 is more complicated and you don't gain > > any benefit of availability. If, in 2, your global heat endpoint is down, > > you can't update the whole stack. You have to work around it by talking > > to Heat (or the individual service endpoints) in the region that is still > > alive. > > Not really, you still have the templates for all of the nested stacks > that are in the remaining region(s). You can manipulate those directly > and not risk getting things out of sync when the Heat master comes back. > > If you really want a global stack, you can recreate it in a different > region and adopt the remaining parts. If there were no dynamic data flowing between regions then the story would be this simple. But we have not forbidden output from one regional template to be used in the input to another. Maybe we should? That would rule out some situations where a user can write something that looks valid but will simply not work (e.g., communicate cross region via wait condition). That would only leave the problem of how the client keeps track of the outputs. You're completely correct of course, there will be a bunch of stuff like that which appears to be possible but actually doesn't work. One advantage of (2) is that the boundary is at least defined, whereas with (4) stuff that doesn't work just looks like any other template. (Wait conditions split across templates don't actually work now, so in that sense this makes it no worse ;) I don't think we want to prohibit it though, because there's a lot of stuff that is legitimate to pass (any data that isn't region-scoped, like public IP addresses). Maybe one day we'll be able to build in some fancy logic to figure out what is valid and what is not. OTOH, the more we restrict what can be done, the less useful this really is. Yes, and if we restrict passing any data, it's probably not going to be useful at all. cheers, Zane. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Heat] Continue discussing multi-region orchestration
Zane Bitter wrote on 11/15/2013 05:59:06 PM: > On 15/11/13 22:17, Keith Bray wrote: > > The way I view 2 vs. 4 is that 2 is more complicated and you don't gain > > any benefit of availability. If, in 2, your global heat endpoint is down, > > you can't update the whole stack. You have to work around it by talking > > to Heat (or the individual service endpoints) in the region that is still > > alive. > > Not really, you still have the templates for all of the nested stacks > that are in the remaining region(s). You can manipulate those directly > and not risk getting things out of sync when the Heat master comes back. > > If you really want a global stack, you can recreate it in a different > region and adopt the remaining parts. If there were no dynamic data flowing between regions then the story would be this simple. But we have not forbidden output from one regional template to be used in the input to another. Maybe we should? That would rule out some situations where a user can write something that looks valid but will simply not work (e.g., communicate cross region via wait condition). That would only leave the problem of how the client keeps track of the outputs. OTOH, the more we restrict what can be done, the less useful this really is. Regards, Mike___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Heat] Continue discussing multi-region orchestration
On 19/11/13 19:03, Clint Byrum wrote: Excerpts from Zane Bitter's message of 2013-11-15 12:41:53 -0800: Good news, everyone! I have created the missing whiteboard diagram that we all needed at the design summit: https://wiki.openstack.org/wiki/Heat/Blueprints/Multi_Region_Support_for_Heat/The_Missing_Diagram I've documented 5 possibilities. (1) is the current implementation, which we agree we want to get away from. I strongly favour (2) for the reasons listed. I don't think (3) has many friends. (4) seems to be popular despite the obvious availability problem and doubts that it is even feasible. Finally, I can save us all some time by simply stating that I will -2 on sight any attempt to implement (5). When we're discussing this, please mention explicitly the number of the model you are talking about at any given time. If you have a suggestion for a different model, make your own diagram! jk, you can sketch it or something for me and I'll see if I can add it. Thanks for putting this together Zane. I just now got around to looking closely. Option 2 is good. I'd love for option 1 to be made automatic by making the client smarter, but parsing templates in the client will require some deep thought before we decide it is a good idea. I'd like to consider a 2a, which just has the same Heat engines the user is talking to being used to do the orchestration in whatever region they are in. I think that is actually the intention of the diagram, but it looks like there is a "special" one that talks to the engines that actually do the work. Yes, I think you're saying the same thing as the diagram was intended to convey. (Each orange box is meant to be a Heat engine.) So the user talks to the Heat endpoint in a region of their choice (doesn't matter which, they're all the same). When a OS::Heat::Stack resource has Region and/or Endpoint specified, it will use python-heatclient to connect to the appropriate engine for that nested stack. 2 may morph into 3 actually, if users don't like the nested stack requirement for 2, we can do the work to basically make the engine create a nested stack per region. So that makes 2 a stronger choice for first implementation. Yeah, if you run your own standalone Heat engine, you've effectively created 3 with no code changes from 2 afaict. I wouldn't recommend that operators deploy this way though. 4 has an unstated pro, which is that attack surface is reduced. This makes more sense when you consider the TripleO case where you may want the undercloud (hardware cloud) to orchestrate things existing in the overcloud (vm cloud) but you don't want the overcloud administrators to be able to control your entire stack. Given CAP theorem, option 5, the global orchestrator, would be doable with not much change as long as partition tolerance were the bit we gave Well, in the real world partitions _do_ happen, so the choice is to give up consistency, availability or both. up. We would just have to have a cross-region RPC bus and database. Of course, since regions are most likely to be partitioned, that is not really a good choice. Trading partition tolerance for consistency lands us in the complexity black hole. Trading out availability makes it no better than option 4. Yep, exactly. cheers, Zane. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Heat] Continue discussing multi-region orchestration
Excerpts from Zane Bitter's message of 2013-11-15 12:41:53 -0800: > Good news, everyone! I have created the missing whiteboard diagram that > we all needed at the design summit: > > https://wiki.openstack.org/wiki/Heat/Blueprints/Multi_Region_Support_for_Heat/The_Missing_Diagram > > I've documented 5 possibilities. (1) is the current implementation, > which we agree we want to get away from. I strongly favour (2) for the > reasons listed. I don't think (3) has many friends. (4) seems to be > popular despite the obvious availability problem and doubts that it is > even feasible. Finally, I can save us all some time by simply stating > that I will -2 on sight any attempt to implement (5). > > When we're discussing this, please mention explicitly the number of the > model you are talking about at any given time. > > If you have a suggestion for a different model, make your own diagram! > jk, you can sketch it or something for me and I'll see if I can add it. Thanks for putting this together Zane. I just now got around to looking closely. Option 2 is good. I'd love for option 1 to be made automatic by making the client smarter, but parsing templates in the client will require some deep thought before we decide it is a good idea. I'd like to consider a 2a, which just has the same Heat engines the user is talking to being used to do the orchestration in whatever region they are in. I think that is actually the intention of the diagram, but it looks like there is a "special" one that talks to the engines that actually do the work. 2 may morph into 3 actually, if users don't like the nested stack requirement for 2, we can do the work to basically make the engine create a nested stack per region. So that makes 2 a stronger choice for first implementation. 4 has an unstated pro, which is that attack surface is reduced. This makes more sense when you consider the TripleO case where you may want the undercloud (hardware cloud) to orchestrate things existing in the overcloud (vm cloud) but you don't want the overcloud administrators to be able to control your entire stack. Given CAP theorem, option 5, the global orchestrator, would be doable with not much change as long as partition tolerance were the bit we gave up. We would just have to have a cross-region RPC bus and database. Of course, since regions are most likely to be partitioned, that is not really a good choice. Trading partition tolerance for consistency lands us in the complexity black hole. Trading out availability makes it no better than option 4. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Heat] Continue discussing multi-region orchestration
Hi, On 11/15/13 9:17 PM, "Georgy Okrokvertskhov" wrote: You are right. At the same time heat feature should not enforce some specific deployment requirements to other openstack components especially taking into account different security considerations. I am trying to rise a concern about possible security implications of that particular approach of using exposed openstack APIs and bring security requirements to the table for discussion. Probably this will help to design better solution for heat to heat or DC to DC communication if it exists. I hope that there is a room for discussion and it is possible to influence on the final design and implementation. I really want this feature to be flexible and useful for most of OpenStack deployments rather then for some specific deployment case.s, Georgy Okrokvertskhov Thank you for your security concerns. There is a room for discussion and possibility to influence the final design. This is the reason why I started this discussion. On 11/15/2013 01:41 PM, Zane Bitter wrote: Good news, everyone! I have created the missing whiteboard diagram that we all needed at the design summit: https://wiki.openstack.org/wiki/Heat/Blueprints/Multi_Region_Support_for_Heat/The_Missing_Diagram I've documented 5 possibilities. (1) is the current implementation, which we agree we want to get away from. I strongly favour (2) for the reasons listed. I don't think (3) has many friends. (4) seems to be popular despite the obvious availability problem and doubts that it is even feasible. Finally, I can save us all some time by simply stating that I will -2 on sight any attempt to implement (5). Zane thank you for creating this diagram! I think it was really the missing piece. Without it people were confused. Right now I think they have better understanding of the possible solutions. On 11/15/2013 01:41 PM, Zane Bitter wrote: On 11/15/13 2:58 PM, "Zane Bitter" wrote: So, to be clear, this is option (4) from the diagram I put together here: https://wiki.openstack.org/wiki/Heat/Blueprints/Multi_Region_Support_for_Heat/The_Missing_Diagram It's got a couple of major problems: * When a whole region goes down, you can lose access to the Heat instance that was managing still-available resources. This makes it more or less impossible to use Heat to manage a highly-available global application. * Instances have to communicate back to the Heat instance that owns them (e.g. for WaitConditions), and it's not yet clear that this is feasible in general. There are also a number of other things I really don't like about this solution (listed on the wiki page), though reasonable people may disagree. cheers, Zane. Yes. You are right. The proof of concept version I implemented is the option (4). I agree with you about the problems you listed. I am not so concerned about the first one (region goes down) but the second one will be a bigger problem in production and I did not notice it before. Also exposing the API endpoint for all the services maybe not the best solution from the security perspective. We probably should go with option (2). On 11/16/2013 09:20 AM, Zane Bitter wrote: On 15/11/13 22:17, Keith Bray wrote: use Heat in that region to do it, you can Adopt the resources into a Heat stack in that region. I don't see how 2 is "Robust against failure of whole region" because if the region on the left part of the picture in 2 goes down, you can't manage your global stack or any of the resources in the left region that are part of that global stack. All you could manage is a subset of resources by manipulating the substack in the right region, but you can do this in 4 as well using Adopt. 4 is a simpler starting use case and easier (IMO) for a user of the system to digest, and has the HUGE advantage of being able to orchestrate deploying resources to multiple regions without a service operator having to have Heat setup and installed in EVERY region. This is particular important for a private cloud/heat person trying to deploy to public cloud.. If doing so requires the public cloud operator have Heat running, then you can't deploy there. If no Heat in that region is required, then you can use your own instance of Heat to deploy to any available openstack cloud. That is a HUGE benefit of 4. Good point, I've added that to the Pro/Con in the wiki. It is the benefit but I think for now we can assume that we have Heat service deployed in each region. The implementation differences between option (2) and (4) does not look like to be big. May be we can add new option in heat configuration file to choose the way to create in different region in case when heat service is not available. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev Best, Bartosz __
Re: [openstack-dev] [Heat] Continue discussing multi-region orchestration
On 16/11/13 00:07, Georgy Okrokvertskhov wrote: Hi, With slight modifications of (2) one can benefit of availability: 1. There should not be a master node. Each heat engine should be able to act as a master if someone asks it to deploy a template. Current master engine will be responsible to contact other engines and pass them the same template and wait for confirmation. 2. Each Heat engine instance receives whole template but deploys only resources which are designated to engine's zone This will provide all benefits of availability as all engines will keep a copy of a template and you can update template by using any heat engine. For example if heat engine in region 1 is down or whole site 1 is down you can update a template with new region settings and update it with using heat engine in region 2. What you're describing is basically Option (5) - build a globally highly-available distributed Heat engine. I don't see how you can describe that as a "slight modification", it's an entire research project that is probably bigger in scope than everything else we have to do in Icehouse put together. As stated previously, I am heavily -2 on this idea. The multi-region support proposal contains information and proposal for "what" and "how" but does not describe "why". If we talk just about multi-region placement support then option (4) works fine. If there is an intension to build a solution for HA, DR, elastic scalability for spike loads then we need to keep availability as a part of a design. I think we should build a system that allows our users to manage highly-available global apps, not try to build one ourselves. cheers, Zane. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Heat] Continue discussing multi-region orchestration
Hi, With slight modifications of (2) one can benefit of availability: 1. There should not be a master node. Each heat engine should be able to act as a master if someone asks it to deploy a template. Current master engine will be responsible to contact other engines and pass them the same template and wait for confirmation. 2. Each Heat engine instance receives whole template but deploys only resources which are designated to engine's zone This will provide all benefits of availability as all engines will keep a copy of a template and you can update template by using any heat engine. For example if heat engine in region 1 is down or whole site 1 is down you can update a template with new region settings and update it with using heat engine in region 2. The multi-region support proposal contains information and proposal for "what" and "how" but does not describe "why". If we talk just about multi-region placement support then option (4) works fine. If there is an intension to build a solution for HA, DR, elastic scalability for spike loads then we need to keep availability as a part of a design. Thanks Georgy On Fri, Nov 15, 2013 at 1:17 PM, Keith Bray wrote: > The way I view 2 vs. 4 is that 2 is more complicated and you don't gain > any benefit of availability. If, in 2, your global heat endpoint is down, > you can't update the whole stack. You have to work around it by talking > to Heat (or the individual service endpoints) in the region that is still > alive. > > 4 is much simpler in that only one Heat instance is involved. If Heat is > down, you still have just as bad/good workaround, which is you talk to > service endpoints in the region that is still available. If you want to > use Heat in that region to do it, you can Adopt the resources into a Heat > stack in that region. I don't see how 2 is "Robust against failure of > whole region" because if the region on the left part of the picture in 2 > goes down, you can't manage your global stack or any of the resources in > the left region that are part of that global stack. All you could manage > is a subset of resources by manipulating the substack in the right region, > but you can do this in 4 as well using Adopt. 4 is a simpler starting use > case and easier (IMO) for a user of the system to digest, and has the HUGE > advantage of being able to orchestrate deploying resources to multiple > regions without a service operator having to have Heat setup and installed > in EVERY region. This is particular important for a private cloud/heat > person trying to deploy to public cloud.. If doing so requires the public > cloud operator have Heat running, then you can't deploy there. If no Heat > in that region is required, then you can use your own instance of Heat to > deploy to any available openstack cloud. That is a HUGE benefit of 4. > > -Keith > > On 11/15/13 2:58 PM, "Zane Bitter" wrote: > > >On 15/11/13 18:24, Bartosz Górski wrote: > >> Hi Thomas, > >> > >> Each of the two engines will be able to create resources in both > >>regions. > >> We do not need to add anything in the heat client. > >> > >> Right now when you want to create a new stack (using heat client or > >> directly API) you need to provide: > >> - stack name > >> - template > >> - parameters (if need) > >> - tenant > >> - username > >> - password > >> - region name (optional) > >> > >> The last four (tenant, username, password and region_name) we will call > >> default context. > >> This context is used in Heat to configure all the openstack clients to > >> other service. > >> Username, password and tenant is used for authentication. > >> Region name is use to get appropriate API endpoint from keystone catalog > >> for other openstack service (like nova). > >> In case with one region you do not need to specify it because there is > >> only one endpoint for each service. > >> In multi-region case we have more than one and region name is used to > >> get correct one. > >> > >> Each nested stack have its own set of openstack clients (nova client, > >> neutron client, ... etc.) inside the heat engine. > >> Right now for all of them the default context is used to configure > >> clients which will be used to create resources. > >> There is not option to change the default context for now. What I'm > >> trying to do it to add possibility to define different > >> context inside the template file. New context can be passed to nested > >> stack resource to create clients set with different > >> endpoints to call. Heat engine will get appropriate endpoints from > >> keystone catalog for specified region name. > >> > >> So from heat engine point of view there is not big change in the > >> workflow. Heat will parse the template, create the > >> dependencies graph and start creating resources in the same way as > >> usual. When he will need to created nested > >> stack with different context he will just use different set of openstack > >> clients (ex. will call services in other re
Re: [openstack-dev] [Heat] Continue discussing multi-region orchestration
On 15/11/13 22:17, Keith Bray wrote: The way I view 2 vs. 4 is that 2 is more complicated and you don't gain any benefit of availability. If, in 2, your global heat endpoint is down, you can't update the whole stack. You have to work around it by talking to Heat (or the individual service endpoints) in the region that is still alive. Not really, you still have the templates for all of the nested stacks that are in the remaining region(s). You can manipulate those directly and not risk getting things out of sync when the Heat master comes back. If you really want a global stack, you can recreate it in a different region and adopt the remaining parts. 4 is much simpler in that only one Heat instance is involved. If Heat is From a code perspective, there is no sense in which (4) is simpler. down, you still have just as bad/good workaround, which is you talk to service endpoints in the region that is still available. If you want to That's considerably worse as a workaround IMO, because it gets reality out of sync with your template. use Heat in that region to do it, you can Adopt the resources into a Heat stack in that region. I don't see how 2 is "Robust against failure of whole region" because if the region on the left part of the picture in 2 goes down, you can't manage your global stack or any of the resources in the left region that are part of that global stack. All you could manage is a subset of resources by manipulating the substack in the right region, but you can do this in 4 as well using Adopt. 4 is a simpler starting use case and easier (IMO) for a user of the system to digest, and has the HUGE advantage of being able to orchestrate deploying resources to multiple regions without a service operator having to have Heat setup and installed in EVERY region. This is particular important for a private cloud/heat person trying to deploy to public cloud.. If doing so requires the public cloud operator have Heat running, then you can't deploy there. If no Heat in that region is required, then you can use your own instance of Heat to deploy to any available openstack cloud. That is a HUGE benefit of 4. Good point, I've added that to the Pro/Con in the wiki. -Keith On 11/15/13 2:58 PM, "Zane Bitter" wrote: On 15/11/13 18:24, Bartosz Górski wrote: Hi Thomas, Each of the two engines will be able to create resources in both regions. We do not need to add anything in the heat client. Right now when you want to create a new stack (using heat client or directly API) you need to provide: - stack name - template - parameters (if need) - tenant - username - password - region name (optional) The last four (tenant, username, password and region_name) we will call default context. This context is used in Heat to configure all the openstack clients to other service. Username, password and tenant is used for authentication. Region name is use to get appropriate API endpoint from keystone catalog for other openstack service (like nova). In case with one region you do not need to specify it because there is only one endpoint for each service. In multi-region case we have more than one and region name is used to get correct one. Each nested stack have its own set of openstack clients (nova client, neutron client, ... etc.) inside the heat engine. Right now for all of them the default context is used to configure clients which will be used to create resources. There is not option to change the default context for now. What I'm trying to do it to add possibility to define different context inside the template file. New context can be passed to nested stack resource to create clients set with different endpoints to call. Heat engine will get appropriate endpoints from keystone catalog for specified region name. So from heat engine point of view there is not big change in the workflow. Heat will parse the template, create the dependencies graph and start creating resources in the same way as usual. When he will need to created nested stack with different context he will just use different set of openstack clients (ex. will call services in other region). So to sum up each of the two heat engine will be able to create resources in both regions if different context will be specified. If only default context will be used heat will create all resource in the same region where it is located. So, to be clear, this is option (4) from the diagram I put together here: https://wiki.openstack.org/wiki/Heat/Blueprints/Multi_Region_Support_for_H eat/The_Missing_Diagram It's got a couple of major problems: * When a whole region goes down, you can lose access to the Heat instance that was managing still-available resources. This makes it more or less impossible to use Heat to manage a highly-available global application. * Instances have to communicate back to the Heat instance that owns them (e.g. for WaitConditions), and it's not yet clear that this is feasible in general. There are also
Re: [openstack-dev] [Heat] Continue discussing multi-region orchestration
The way I view 2 vs. 4 is that 2 is more complicated and you don't gain any benefit of availability. If, in 2, your global heat endpoint is down, you can't update the whole stack. You have to work around it by talking to Heat (or the individual service endpoints) in the region that is still alive. 4 is much simpler in that only one Heat instance is involved. If Heat is down, you still have just as bad/good workaround, which is you talk to service endpoints in the region that is still available. If you want to use Heat in that region to do it, you can Adopt the resources into a Heat stack in that region. I don't see how 2 is "Robust against failure of whole region" because if the region on the left part of the picture in 2 goes down, you can't manage your global stack or any of the resources in the left region that are part of that global stack. All you could manage is a subset of resources by manipulating the substack in the right region, but you can do this in 4 as well using Adopt. 4 is a simpler starting use case and easier (IMO) for a user of the system to digest, and has the HUGE advantage of being able to orchestrate deploying resources to multiple regions without a service operator having to have Heat setup and installed in EVERY region. This is particular important for a private cloud/heat person trying to deploy to public cloud.. If doing so requires the public cloud operator have Heat running, then you can't deploy there. If no Heat in that region is required, then you can use your own instance of Heat to deploy to any available openstack cloud. That is a HUGE benefit of 4. -Keith On 11/15/13 2:58 PM, "Zane Bitter" wrote: >On 15/11/13 18:24, Bartosz Górski wrote: >> Hi Thomas, >> >> Each of the two engines will be able to create resources in both >>regions. >> We do not need to add anything in the heat client. >> >> Right now when you want to create a new stack (using heat client or >> directly API) you need to provide: >> - stack name >> - template >> - parameters (if need) >> - tenant >> - username >> - password >> - region name (optional) >> >> The last four (tenant, username, password and region_name) we will call >> default context. >> This context is used in Heat to configure all the openstack clients to >> other service. >> Username, password and tenant is used for authentication. >> Region name is use to get appropriate API endpoint from keystone catalog >> for other openstack service (like nova). >> In case with one region you do not need to specify it because there is >> only one endpoint for each service. >> In multi-region case we have more than one and region name is used to >> get correct one. >> >> Each nested stack have its own set of openstack clients (nova client, >> neutron client, ... etc.) inside the heat engine. >> Right now for all of them the default context is used to configure >> clients which will be used to create resources. >> There is not option to change the default context for now. What I'm >> trying to do it to add possibility to define different >> context inside the template file. New context can be passed to nested >> stack resource to create clients set with different >> endpoints to call. Heat engine will get appropriate endpoints from >> keystone catalog for specified region name. >> >> So from heat engine point of view there is not big change in the >> workflow. Heat will parse the template, create the >> dependencies graph and start creating resources in the same way as >> usual. When he will need to created nested >> stack with different context he will just use different set of openstack >> clients (ex. will call services in other region). >> >> So to sum up each of the two heat engine will be able to create >> resources in both regions if different context will >> be specified. If only default context will be used heat will create all >> resource in the same region where it is located. > >So, to be clear, this is option (4) from the diagram I put together here: >https://wiki.openstack.org/wiki/Heat/Blueprints/Multi_Region_Support_for_H >eat/The_Missing_Diagram > >It's got a couple of major problems: > >* When a whole region goes down, you can lose access to the Heat >instance that was managing still-available resources. This makes it more >or less impossible to use Heat to manage a highly-available global >application. > >* Instances have to communicate back to the Heat instance that owns them >(e.g. for WaitConditions), and it's not yet clear that this is feasible >in general. > >There are also a number of other things I really don't like about this >solution (listed on the wiki page), though reasonable people may disagree. > >cheers, >Zane. > >___ >OpenStack-dev mailing list >OpenStack-dev@lists.openstack.org >http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lis
Re: [openstack-dev] [Heat] Continue discussing multi-region orchestration
On 15/11/13 18:24, Bartosz Górski wrote: Hi Thomas, Each of the two engines will be able to create resources in both regions. We do not need to add anything in the heat client. Right now when you want to create a new stack (using heat client or directly API) you need to provide: - stack name - template - parameters (if need) - tenant - username - password - region name (optional) The last four (tenant, username, password and region_name) we will call default context. This context is used in Heat to configure all the openstack clients to other service. Username, password and tenant is used for authentication. Region name is use to get appropriate API endpoint from keystone catalog for other openstack service (like nova). In case with one region you do not need to specify it because there is only one endpoint for each service. In multi-region case we have more than one and region name is used to get correct one. Each nested stack have its own set of openstack clients (nova client, neutron client, ... etc.) inside the heat engine. Right now for all of them the default context is used to configure clients which will be used to create resources. There is not option to change the default context for now. What I'm trying to do it to add possibility to define different context inside the template file. New context can be passed to nested stack resource to create clients set with different endpoints to call. Heat engine will get appropriate endpoints from keystone catalog for specified region name. So from heat engine point of view there is not big change in the workflow. Heat will parse the template, create the dependencies graph and start creating resources in the same way as usual. When he will need to created nested stack with different context he will just use different set of openstack clients (ex. will call services in other region). So to sum up each of the two heat engine will be able to create resources in both regions if different context will be specified. If only default context will be used heat will create all resource in the same region where it is located. So, to be clear, this is option (4) from the diagram I put together here: https://wiki.openstack.org/wiki/Heat/Blueprints/Multi_Region_Support_for_Heat/The_Missing_Diagram It's got a couple of major problems: * When a whole region goes down, you can lose access to the Heat instance that was managing still-available resources. This makes it more or less impossible to use Heat to manage a highly-available global application. * Instances have to communicate back to the Heat instance that owns them (e.g. for WaitConditions), and it's not yet clear that this is feasible in general. There are also a number of other things I really don't like about this solution (listed on the wiki page), though reasonable people may disagree. cheers, Zane. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Heat] Continue discussing multi-region orchestration
On 11/15/2013 01:41 PM, Zane Bitter wrote: Good news, everyone! I have created the missing whiteboard diagram that we all needed at the design summit: https://wiki.openstack.org/wiki/Heat/Blueprints/Multi_Region_Support_for_Heat/The_Missing_Diagram I've documented 5 possibilities. (1) is the current implementation, which we agree we want to get away from. I strongly favour (2) for the reasons listed. I don't think (3) has many friends. (4) seems to be popular despite the obvious availability problem and doubts that it is even feasible. Finally, I can save us all some time by simply stating that I will -2 on sight any attempt to implement (5). 1 status quo isn't terrible 2 +2 simple robust 3 -2 this means another daemon and seems overly complex 4 I don't think it will be possible to allow VM->heat communication for things like signaling. Without signaling, this is DOA 5 seems complex to implement and I don't see a clear path to get there from where we are today Out of the options, #2 looks like the best choice. I would -2 option #3 if I saw it hit the review queue. Regards -steve When we're discussing this, please mention explicitly the number of the model you are talking about at any given time. If you have a suggestion for a different model, make your own diagram! jk, you can sketch it or something for me and I'll see if I can add it. cheers, Zane. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Heat] Continue discussing multi-region orchestration
Good news, everyone! I have created the missing whiteboard diagram that we all needed at the design summit: https://wiki.openstack.org/wiki/Heat/Blueprints/Multi_Region_Support_for_Heat/The_Missing_Diagram I've documented 5 possibilities. (1) is the current implementation, which we agree we want to get away from. I strongly favour (2) for the reasons listed. I don't think (3) has many friends. (4) seems to be popular despite the obvious availability problem and doubts that it is even feasible. Finally, I can save us all some time by simply stating that I will -2 on sight any attempt to implement (5). When we're discussing this, please mention explicitly the number of the model you are talking about at any given time. If you have a suggestion for a different model, make your own diagram! jk, you can sketch it or something for me and I'll see if I can add it. cheers, Zane. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Heat] Continue discussing multi-region orchestration
On Fri, Nov 15, 2013 at 10:44 AM, Stephen Gran wrote: > > > Surely those are local deployment policy decisions that shouldn't affect > the development of capabilities in heat itself, right? If a deployer does > not want one heat deployment to be able to reach some endpoints, they'll > set up a local heat that can reach those endpoints and deploy their stack > through that one, right? > > You are right. At the same time heat feature should not enforce some specific deployment requirements to other openstack components especially taking into account different security considerations. I am trying to rise a concern about possible security implications of that particular approach of using exposed openstack APIs and bring security requirements to the table for discussion. Probably this will help to design better solution for heat to heat or DC to DC communication if it exists. I hope that there is a room for discussion and it is possible to influence on the final design and implementation. I really want this feature to be flexible and useful for most of OpenStack deployments rather then for some specific deployment case. -- Georgy Okrokvertskhov Technical Program Manager, Cloud and Infrastructure Services, Mirantis http://www.mirantis.com Tel. +1 650 963 9828 Mob. +1 650 996 3284 ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Heat] Continue discussing multi-region orchestration
On 11/15/2013 12:24 PM, Bartosz Górski wrote: > Hi Thomas, > > Each of the two engines will be able to create resources in both regions. > We do not need to add anything in the heat client. > > Right now when you want to create a new stack (using heat client or > directly API) you need to provide: > - stack name > - template > - parameters (if need) > - tenant > - username > - password > - region name (optional) > > The last four (tenant, username, password and region_name) we will call > default context. > This context is used in Heat to configure all the openstack clients to > other service. > Username, password and tenant is used for authentication. > Region name is use to get appropriate API endpoint from keystone catalog > for other openstack service (like nova). > In case with one region you do not need to specify it because there is > only one endpoint for each service. > In multi-region case we have more than one and region name is used to > get correct one. > > Each nested stack have its own set of openstack clients (nova client, > neutron client, ... etc.) inside the heat engine. > Right now for all of them the default context is used to configure > clients which will be used to create resources. > There is not option to change the default context for now. What I'm > trying to do it to add possibility to define different > context inside the template file. New context can be passed to nested > stack resource to create clients set with different > endpoints to call. Heat engine will get appropriate endpoints from > keystone catalog for specified region name. > > So from heat engine point of view there is not big change in the > workflow. Heat will parse the template, create the > dependencies graph and start creating resources in the same way as > usual. When he will need to created nested > stack with different context he will just use different set of openstack > clients (ex. will call services in other region). > > So to sum up each of the two heat engine will be able to create > resources in both regions if different context will > be specified. If only default context will be used heat will create all > resource in the same region where it is located. One more step with the above and you might get to where I want you to be, which is to be able to register an additional context myself, that itself might have a specified endpoint, username, tenant and password. Yup - so that I could run a heat in RAX and use that heat to orchestrate resources in RAX and HP. WOOT > On 11/15/2013 08:19 AM, Thomas Spatzier wrote: >> Hi Bartosz, >> >> one thing that is still not clear to me: >> In the discussion at the summit and in your updated architecture on the >> wiki it talks about two Heat engines, one in each region, and on-top only >> the dashboard. So what entity will really do the cross region >> orchestration? In the discussions I heard people talking about the heat >> client doing it. But wouldn't that duplicate functionality that is inside >> the engine into the client, i.e. the client would become an orchestrator >> itself then? >> >> Regards, >> Thomas >> >> Bartosz Górski wrote on 14.11.2013 15:58:39: >>> From: Bartosz Górski >>> To: openstack-dev@lists.openstack.org, >>> Date: 14.11.2013 16:05 >>> Subject: [openstack-dev] [Heat] Continue discussing multi-region >> orchestration >>> Hi all, >>> >>> At summit in Hong Kong we had a design session where we discussed adding >>> multi-region orchestration support to Heat. During the session we had >>> really heated discussion and spent most of the time on explaining the >>> problem. I think it was really good starting point and right now more >>> people have better understanding for this problem. I appreciate all the >>> suggestions and concerns I got from you. I would like to continue this >>> discussion here on the mailing list. >>> >>> I updated the etherpad after the session. If I forgot about something or >>> wrote something that is not right feel free a please tell me about it. >>> >>> References: >>> [1] Blueprint: >>> >> https://wiki.openstack.org/wiki/Heat/Blueprints/Multi_Region_Support_for_Heat >> >> >>> [2] Etherpad: >>> https://etherpad.openstack.org/p/icehouse-summit-heat-multi-region-cloud >>> [3] Patch with POC version: https://review.openstack.org/#/c/53313/ >>> >>> >>> Best, >>> Bartosz Górski >>> NTTi3 >>> >>> >>>
Re: [openstack-dev] [Heat] Continue discussing multi-region orchestration
On 15/11/13 18:35, Georgy Okrokvertskhov wrote: First of all, I believe this approach assumes that heat engine can reach API endpoints which are located in different region. I am not sure if it is a default deployment approach when someone is exposing OpenStack services endpoints to the outside of the local DC network. The same problem occurs with communication from VM to Heat engine. Heat passes IP address for cloud watch, ceilometer and for signaling in order to use waitcoinditions. So do you expect that VM will be able to communicate with Heat engine who is in different location? That will assume that networks in both locations are tightly coupled which is not the common case I believe. Surely those are local deployment policy decisions that shouldn't affect the development of capabilities in heat itself, right? If a deployer does not want one heat deployment to be able to reach some endpoints, they'll set up a local heat that can reach those endpoints and deploy their stack through that one, right? Cheers, -- Stephen Gran Senior Systems Integrator - theguardian.com Please consider the environment before printing this email. -- Visit theguardian.com On your mobile, download the Guardian iPhone app theguardian.com/iphone and our iPad edition theguardian.com/iPad Save up to 33% by subscribing to the Guardian and Observer - choose the papers you want and get full digital access. Visit subscribe.theguardian.com This e-mail and all attachments are confidential and may also be privileged. If you are not the named recipient, please notify the sender and delete the e-mail and all attachments immediately. Do not disclose the contents to another person. You may not use the information for any purpose, or store, or copy, it in any way. Guardian News & Media Limited is not liable for any computer viruses or other material transmitted with or as part of this e-mail. You should employ virus checking software. Guardian News & Media Limited A member of Guardian Media Group plc Registered Office PO Box 68164 Kings Place 90 York Way London N1P 2AP Registered in England Number 908396 -- ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Heat] Continue discussing multi-region orchestration
Hi Bartosz, Could you please clarify the section about resource creation in other regions via calling their endpoints? >So from heat engine point of view there is not big change in the workflow. Heat will parse the template, >create the >dependencies graph and start creating resources in the same way as usual. When he will need to created >nested >stack with different context he will just use different set of openstack clients (ex. will call services in other >region). >So to sum up each of the two heat engine will be able to create resources in both regions if different context >will be specified. If only default context will be used heat will create all resource in the same region where it >is located. First of all, I believe this approach assumes that heat engine can reach API endpoints which are located in different region. I am not sure if it is a default deployment approach when someone is exposing OpenStack services endpoints to the outside of the local DC network. The same problem occurs with communication from VM to Heat engine. Heat passes IP address for cloud watch, ceilometer and for signaling in order to use waitcoinditions. So do you expect that VM will be able to communicate with Heat engine who is in different location? That will assume that networks in both locations are tightly coupled which is not the common case I believe. Thanks Georgy On Fri, Nov 15, 2013 at 9:24 AM, Bartosz Górski wrote: > Hi Thomas, > > Each of the two engines will be able to create resources in both regions. > We do not need to add anything in the heat client. > > Right now when you want to create a new stack (using heat client or > directly API) you need to provide: > - stack name > - template > - parameters (if need) > - tenant > - username > - password > - region name (optional) > > The last four (tenant, username, password and region_name) we will call > default context. > This context is used in Heat to configure all the openstack clients to > other service. > Username, password and tenant is used for authentication. > Region name is use to get appropriate API endpoint from keystone catalog > for other openstack service (like nova). > In case with one region you do not need to specify it because there is > only one endpoint for each service. > In multi-region case we have more than one and region name is used to get > correct one. > > Each nested stack have its own set of openstack clients (nova client, > neutron client, ... etc.) inside the heat engine. > Right now for all of them the default context is used to configure clients > which will be used to create resources. > There is not option to change the default context for now. What I'm trying > to do it to add possibility to define different > context inside the template file. New context can be passed to nested > stack resource to create clients set with different > endpoints to call. Heat engine will get appropriate endpoints from > keystone catalog for specified region name. > > So from heat engine point of view there is not big change in the workflow. > Heat will parse the template, create the > dependencies graph and start creating resources in the same way as usual. > When he will need to created nested > stack with different context he will just use different set of openstack > clients (ex. will call services in other region). > > So to sum up each of the two heat engine will be able to create resources > in both regions if different context will > be specified. If only default context will be used heat will create all > resource in the same region where it is located. > > Best, > Bartosz > > > > On 11/15/2013 08:19 AM, Thomas Spatzier wrote: > >> Hi Bartosz, >> >> one thing that is still not clear to me: >> In the discussion at the summit and in your updated architecture on the >> wiki it talks about two Heat engines, one in each region, and on-top only >> the dashboard. So what entity will really do the cross region >> orchestration? In the discussions I heard people talking about the heat >> client doing it. But wouldn't that duplicate functionality that is inside >> the engine into the client, i.e. the client would become an orchestrator >> itself then? >> >> Regards, >> Thomas >> >> Bartosz Górski wrote on 14.11.2013 15:58:39: >> >>> From: Bartosz Górski >>> To: openstack-dev@lists.openstack.org, >>> Date: 14.11.2013 16:05 >>> Subject: [openstack-dev] [Heat] Continue discussing multi-region >>> >> orchestration >> >>> Hi all, >>> >>> At summit in Hong Kong we had a design session where we discussed adding >>> multi-region orchestration support
Re: [openstack-dev] [Heat] Continue discussing multi-region orchestration
Hi Thomas, Each of the two engines will be able to create resources in both regions. We do not need to add anything in the heat client. Right now when you want to create a new stack (using heat client or directly API) you need to provide: - stack name - template - parameters (if need) - tenant - username - password - region name (optional) The last four (tenant, username, password and region_name) we will call default context. This context is used in Heat to configure all the openstack clients to other service. Username, password and tenant is used for authentication. Region name is use to get appropriate API endpoint from keystone catalog for other openstack service (like nova). In case with one region you do not need to specify it because there is only one endpoint for each service. In multi-region case we have more than one and region name is used to get correct one. Each nested stack have its own set of openstack clients (nova client, neutron client, ... etc.) inside the heat engine. Right now for all of them the default context is used to configure clients which will be used to create resources. There is not option to change the default context for now. What I'm trying to do it to add possibility to define different context inside the template file. New context can be passed to nested stack resource to create clients set with different endpoints to call. Heat engine will get appropriate endpoints from keystone catalog for specified region name. So from heat engine point of view there is not big change in the workflow. Heat will parse the template, create the dependencies graph and start creating resources in the same way as usual. When he will need to created nested stack with different context he will just use different set of openstack clients (ex. will call services in other region). So to sum up each of the two heat engine will be able to create resources in both regions if different context will be specified. If only default context will be used heat will create all resource in the same region where it is located. Best, Bartosz On 11/15/2013 08:19 AM, Thomas Spatzier wrote: Hi Bartosz, one thing that is still not clear to me: In the discussion at the summit and in your updated architecture on the wiki it talks about two Heat engines, one in each region, and on-top only the dashboard. So what entity will really do the cross region orchestration? In the discussions I heard people talking about the heat client doing it. But wouldn't that duplicate functionality that is inside the engine into the client, i.e. the client would become an orchestrator itself then? Regards, Thomas Bartosz Górski wrote on 14.11.2013 15:58:39: From: Bartosz Górski To: openstack-dev@lists.openstack.org, Date: 14.11.2013 16:05 Subject: [openstack-dev] [Heat] Continue discussing multi-region orchestration Hi all, At summit in Hong Kong we had a design session where we discussed adding multi-region orchestration support to Heat. During the session we had really heated discussion and spent most of the time on explaining the problem. I think it was really good starting point and right now more people have better understanding for this problem. I appreciate all the suggestions and concerns I got from you. I would like to continue this discussion here on the mailing list. I updated the etherpad after the session. If I forgot about something or wrote something that is not right feel free a please tell me about it. References: [1] Blueprint: https://wiki.openstack.org/wiki/Heat/Blueprints/Multi_Region_Support_for_Heat [2] Etherpad: https://etherpad.openstack.org/p/icehouse-summit-heat-multi-region-cloud [3] Patch with POC version: https://review.openstack.org/#/c/53313/ Best, Bartosz Górski NTTi3 ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Heat] Continue discussing multi-region orchestration
Hi Bartosz, one thing that is still not clear to me: In the discussion at the summit and in your updated architecture on the wiki it talks about two Heat engines, one in each region, and on-top only the dashboard. So what entity will really do the cross region orchestration? In the discussions I heard people talking about the heat client doing it. But wouldn't that duplicate functionality that is inside the engine into the client, i.e. the client would become an orchestrator itself then? Regards, Thomas Bartosz Górski wrote on 14.11.2013 15:58:39: > From: Bartosz Górski > To: openstack-dev@lists.openstack.org, > Date: 14.11.2013 16:05 > Subject: [openstack-dev] [Heat] Continue discussing multi-region orchestration > > Hi all, > > At summit in Hong Kong we had a design session where we discussed adding > multi-region orchestration support to Heat. During the session we had > really heated discussion and spent most of the time on explaining the > problem. I think it was really good starting point and right now more > people have better understanding for this problem. I appreciate all the > suggestions and concerns I got from you. I would like to continue this > discussion here on the mailing list. > > I updated the etherpad after the session. If I forgot about something or > wrote something that is not right feel free a please tell me about it. > > References: > [1] Blueprint: > https://wiki.openstack.org/wiki/Heat/Blueprints/Multi_Region_Support_for_Heat > [2] Etherpad: > https://etherpad.openstack.org/p/icehouse-summit-heat-multi-region-cloud > [3] Patch with POC version: https://review.openstack.org/#/c/53313/ > > > Best, > Bartosz Górski > NTTi3 > > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [Heat] Continue discussing multi-region orchestration
Hi all, At summit in Hong Kong we had a design session where we discussed adding multi-region orchestration support to Heat. During the session we had really heated discussion and spent most of the time on explaining the problem. I think it was really good starting point and right now more people have better understanding for this problem. I appreciate all the suggestions and concerns I got from you. I would like to continue this discussion here on the mailing list. I updated the etherpad after the session. If I forgot about something or wrote something that is not right feel free a please tell me about it. References: [1] Blueprint: https://wiki.openstack.org/wiki/Heat/Blueprints/Multi_Region_Support_for_Heat [2] Etherpad: https://etherpad.openstack.org/p/icehouse-summit-heat-multi-region-cloud [3] Patch with POC version: https://review.openstack.org/#/c/53313/ Best, Bartosz Górski NTTi3 ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
