Re: [openstack-dev] [Solum] Separation of concerns for 0.1 git deploy blueprint
Hi Adrian Sounds like there's a need for a true workflow engine here. I suspect that for any use case beyond trivial, each user will have her own flavor of CD (TripleO is a good example for this) - some will prefer to use a canary server and test it manually before deploying to other servers, some will deploy to specific servers based on the subset of users they're serving, some will do a red/black type of deployment (Which maps well into environments below), etc. But I think that there should be an user-configurable mechanism to define and tweak these workflows. There will also be additional related workflows - e.g. rolling back a deployment, or updating the infrastructure components of a running application (e.g. database version upgrade). I might be raising a topic that's already been discussed (sorry if I am), but I was wondering how this would fit into the architecture. Thanks! Uri On Oct 31, 2013, at 6:42 AM, Adrian Otto adrian.o...@rackspace.com wrote: Robert, We would be interested in exploring your use case for CD, and help you judge what would be the best fit. I know that Monty Taylor has identified some parts of openstack-infra that we could potentially leverage in Solum to address our CI/CD goals. The design of that feature is still pending, so we are looking forward to sourcing its of input on that. I don't anticipate any fundament gaps that would prevent you from using a future version of Solum to help narrow your scope. The whole point of our focus is to try to make these goals easier to achieve, and try to get maximum leverage of what the OpenStack ecosystem already offers. The high level concept behind the GIt Integration feature currently conceived for Solum is to offer application developers a way to commit code to a Git repo and upon a git push command, have a remote system trigger a modular CI/CD pipeline that ends with that application running on an OpenStack cloud in one or more environments (dev/test/stage/prod/etc.). In a simple case that may be (in the case of a dynamic language like Python) the testing of the code, arrangement of the commit into a deployable bundle, triggering of the Solum API to trigger a deployment, automatic generation of a HOT artifact, and stimulation of the Heat API and any other API calls needed. In a more complex scenario, you may want additional features and capabilities, such as a system like Gerrit to trigger the commit/push, integrate an artifact repository for storing the builds, etc. Using Zuul for that would be appropriate for sure. There are also dev shops who already have a lot invested into their Jenkins setups, and will want to bypass portions of what I described above, so there will be a use case supported where you can just integrate with the solum API, and pass in a pre-built bundle artifact and bypass part or all of Solum's CI features, and just use it for deployment and re-deployment and promotion of a deployment between various environments (dev/test/pros/etc.). You would expect features like canary deploy in that case that might make it more attractive than just generating your own HOT artifact in your Jenkins workflow and passing that in directly for orchestration. You should expect a higher level of control for management of the deployed system with abstracts that let you easily extend the system for monitoring and management reasons. Note that not all this functionality belongs in Solum. Some of it may actually come from Heat, and other downstream systems. The idea is that we address the key gaps. Adrian On Oct 30, 2013, at 9:16 PM, Robert Collins robe...@robertcollins.net wrote: Hi Adrian, I'm very interested in Solum, particularly as it relates to TripleO : as you know the TripleO story is to treat OpenStack as just another application to install via Heat/Nova etc - and we've got a major story around pulling from git - CI + CD pipeline - automated deployment to production. So - where we can say 'hey, this should be part of Solum', we can hive stuff out of the TripleO program into Solum - but only if we've got compatible stories. Right now, our design for CD is basically Gerrit + Zuul - exactly the same as gating checks for OpenStack itself. When I look at https://wiki.openstack.org/wiki/Solum/FeatureBlueprints/GitIntegration#Flow_Boundaries I'm having a little trouble mapping that into your design. I'm wondering if you can expand on how that hangs together? Thanks! Rob On 31 October 2013 14:01, Adrian Otto adrian.o...@rackspace.com wrote: Clayton, Thanks for adding the diagram illustrating the flow. I expect that respond to client may not be a synchronous flow, rather that if creation of a repo takes a while that the API may return a 202 Accepted, and the client can poll a status attribute to determine completion and learn the actual location of the repo created. So in that case
Re: [openstack-dev] [Solum] Separation of concerns for 0.1 git deploy blueprint
Uri, Yes. We will need to offer a facility for customizing the CD pipeline(s). We have not contemplated all of the possibilities for that yet, so this is something that we will certainly discuss further. We will need to strike the right balance between options and simplicity. We have thought through the possibility of generating a HOT[1] artifact in the case that one is not provided, and allowing a user-supplied one in the event that one is supplied with the code at the time of the deployment action. Of course if no HOT is provided, we can generate one (or a simple hierarchy of them). That option would allow you to cause anything that Heat knows how to do. We will continue to contribute to Heat (and other downstream projects) to help it achieve its mission. From my view, Heat should allow us to connect various different imperative systems downstream. For example, if you want something to happen on a given node type, you might use Chef or Puppet to accomplish that. Features related to orchestration (the things offered in the HOT as declarations) should be provided by heat, or one of its resource types. That's where I see most features like rollback fitting in. The actual design of the logic that allows the promotion of a deployment through different Environments[2] will likely be triggered by Solum, but implemented downstream in Heat. A system level rollback (user requested, rather than initiated by a failure) may actually be achieved by a request to swap Environments, or modify one. Adrian References: [1]: https://wiki.openstack.org/wiki/Heat/DSL#Heat_Orchestration_Template_.28HOT.29 [2]: https://wiki.openstack.org/wiki/Heat/DSL#Environment On Oct 31, 2013, at 2:04 AM, Uri Cohen u...@gigaspaces.com wrote: Hi Adrian Sounds like there's a need for a true workflow engine here. I suspect that for any use case beyond trivial, each user will have her own flavor of CD (TripleO is a good example for this) - some will prefer to use a canary server and test it manually before deploying to other servers, some will deploy to specific servers based on the subset of users they're serving, some will do a red/black type of deployment (Which maps well into environments below), etc. But I think that there should be an user-configurable mechanism to define and tweak these workflows. There will also be additional related workflows - e.g. rolling back a deployment, or updating the infrastructure components of a running application (e.g. database version upgrade). I might be raising a topic that's already been discussed (sorry if I am), but I was wondering how this would fit into the architecture. Thanks! Uri On Oct 31, 2013, at 6:42 AM, Adrian Otto adrian.o...@rackspace.com wrote: Robert, We would be interested in exploring your use case for CD, and help you judge what would be the best fit. I know that Monty Taylor has identified some parts of openstack-infra that we could potentially leverage in Solum to address our CI/CD goals. The design of that feature is still pending, so we are looking forward to sourcing its of input on that. I don't anticipate any fundament gaps that would prevent you from using a future version of Solum to help narrow your scope. The whole point of our focus is to try to make these goals easier to achieve, and try to get maximum leverage of what the OpenStack ecosystem already offers. The high level concept behind the GIt Integration feature currently conceived for Solum is to offer application developers a way to commit code to a Git repo and upon a git push command, have a remote system trigger a modular CI/CD pipeline that ends with that application running on an OpenStack cloud in one or more environments (dev/test/stage/prod/etc.). In a simple case that may be (in the case of a dynamic language like Python) the testing of the code, arrangement of the commit into a deployable bundle, triggering of the Solum API to trigger a deployment, automatic generation of a HOT artifact, and stimulation of the Heat API and any other API calls needed. In a more complex scenario, you may want additional features and capabilities, such as a system like Gerrit to trigger the commit/push, integrate an artifact repository for storing the builds, etc. Using Zuul for that would be appropriate for sure. There are also dev shops who already have a lot invested into their Jenkins setups, and will want to bypass portions of what I described above, so there will be a use case supported where you can just integrate with the solum API, and pass in a pre-built bundle artifact and bypass part or all of Solum's CI features, and just use it for deployment and re-deployment and promotion of a deployment between various environments (dev/test/pros/etc.). You would expect features like canary deploy in that case that might make it more attractive than just generating your own HOT artifact in your
Re: [openstack-dev] [Solum] Separation of concerns for 0.1 git deploy blueprint
Clayton, Thanks for adding the diagram illustrating the flow. I expect that respond to client may not be a synchronous flow, rather that if creation of a repo takes a while that the API may return a 202 Accepted, and the client can poll a status attribute to determine completion and learn the actual location of the repo created. So in that case respond to client also might mean update async status. This will be particularly important in cases where an external SCM system is used (perhaps Github). I agree that this will be a helpful reference to guide our upcoming interactive design sessions. I have created the first call for participation, which will also be separately announced: https://wiki.openstack.org/wiki/Solum/BreakoutMeetings Adrian On Oct 30, 2013, at 2:45 PM, Clayton Coleman ccole...@redhat.com wrote: In the IRC meeting yesterday [1] we discussed splitting the individual topics for the git deploy blueprint [2] into rough sub areas. To help frame the abstractions we've been discussing I roughed out a flow based on two user inputs, REST API create and a git push and then tried to draw boxes around the related bits. The abstractions roughly correspond to the potential areas that can be the focus of break out discussions (and if they don't a good discussion of why is always helpful). Squiggly boxes are potential plugin points and/or strong responsibility boundaries. Feedback desirable. [1] http://irclogs.solum.io/2013/solum.2013-10-29-16.01.txt [2] https://wiki.openstack.org/wiki/Solum/FeatureBlueprints/GitIntegration#Flow_Boundaries ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Solum] Separation of concerns for 0.1 git deploy blueprint
Hi Adrian, I'm very interested in Solum, particularly as it relates to TripleO : as you know the TripleO story is to treat OpenStack as just another application to install via Heat/Nova etc - and we've got a major story around pulling from git - CI + CD pipeline - automated deployment to production. So - where we can say 'hey, this should be part of Solum', we can hive stuff out of the TripleO program into Solum - but only if we've got compatible stories. Right now, our design for CD is basically Gerrit + Zuul - exactly the same as gating checks for OpenStack itself. When I look at https://wiki.openstack.org/wiki/Solum/FeatureBlueprints/GitIntegration#Flow_Boundaries I'm having a little trouble mapping that into your design. I'm wondering if you can expand on how that hangs together? Thanks! Rob On 31 October 2013 14:01, Adrian Otto adrian.o...@rackspace.com wrote: Clayton, Thanks for adding the diagram illustrating the flow. I expect that respond to client may not be a synchronous flow, rather that if creation of a repo takes a while that the API may return a 202 Accepted, and the client can poll a status attribute to determine completion and learn the actual location of the repo created. So in that case respond to client also might mean update async status. This will be particularly important in cases where an external SCM system is used (perhaps Github). I agree that this will be a helpful reference to guide our upcoming interactive design sessions. I have created the first call for participation, which will also be separately announced: https://wiki.openstack.org/wiki/Solum/BreakoutMeetings Adrian On Oct 30, 2013, at 2:45 PM, Clayton Coleman ccole...@redhat.com wrote: In the IRC meeting yesterday [1] we discussed splitting the individual topics for the git deploy blueprint [2] into rough sub areas. To help frame the abstractions we've been discussing I roughed out a flow based on two user inputs, REST API create and a git push and then tried to draw boxes around the related bits. The abstractions roughly correspond to the potential areas that can be the focus of break out discussions (and if they don't a good discussion of why is always helpful). Squiggly boxes are potential plugin points and/or strong responsibility boundaries. Feedback desirable. [1] http://irclogs.solum.io/2013/solum.2013-10-29-16.01.txt [2] https://wiki.openstack.org/wiki/Solum/FeatureBlueprints/GitIntegration#Flow_Boundaries ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Robert Collins rbtcoll...@hp.com Distinguished Technologist HP Converged Cloud ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Solum] Separation of concerns for 0.1 git deploy blueprint
Robert, We would be interested in exploring your use case for CD, and help you judge what would be the best fit. I know that Monty Taylor has identified some parts of openstack-infra that we could potentially leverage in Solum to address our CI/CD goals. The design of that feature is still pending, so we are looking forward to sourcing its of input on that. I don't anticipate any fundament gaps that would prevent you from using a future version of Solum to help narrow your scope. The whole point of our focus is to try to make these goals easier to achieve, and try to get maximum leverage of what the OpenStack ecosystem already offers. The high level concept behind the GIt Integration feature currently conceived for Solum is to offer application developers a way to commit code to a Git repo and upon a git push command, have a remote system trigger a modular CI/CD pipeline that ends with that application running on an OpenStack cloud in one or more environments (dev/test/stage/prod/etc.). In a simple case that may be (in the case of a dynamic language like Python) the testing of the code, arrangement of the commit into a deployable bundle, triggering of the Solum API to trigger a deployment, automatic generation of a HOT artifact, and stimulation of the Heat API and any other API calls needed. In a more complex scenario, you may want additional features and capabilities, such as a system like Gerrit to trigger the commit/push, integrate an artifact repository for storing the builds, etc. Using Zuul for that would be appropriate for sure. There are also dev shops who already have a lot invested into their Jenkins setups, and will want to bypass portions of what I described above, so there will be a use case supported where you can just integrate with the solum API, and pass in a pre-built bundle artifact and bypass part or all of Solum's CI features, and just use it for deployment and re-deployment and promotion of a deployment between various environments (dev/test/pros/etc.). You would expect features like canary deploy in that case that might make it more attractive than just generating your own HOT artifact in your Jenkins workflow and passing that in directly for orchestration. You should expect a higher level of control for management of the deployed system with abstracts that let you easily extend the system for monitoring and management reasons. Note that not all this functionality belongs in Solum. Some of it may actually come from Heat, and other downstream systems. The idea is that we address the key gaps. Adrian On Oct 30, 2013, at 9:16 PM, Robert Collins robe...@robertcollins.net wrote: Hi Adrian, I'm very interested in Solum, particularly as it relates to TripleO : as you know the TripleO story is to treat OpenStack as just another application to install via Heat/Nova etc - and we've got a major story around pulling from git - CI + CD pipeline - automated deployment to production. So - where we can say 'hey, this should be part of Solum', we can hive stuff out of the TripleO program into Solum - but only if we've got compatible stories. Right now, our design for CD is basically Gerrit + Zuul - exactly the same as gating checks for OpenStack itself. When I look at https://wiki.openstack.org/wiki/Solum/FeatureBlueprints/GitIntegration#Flow_Boundaries I'm having a little trouble mapping that into your design. I'm wondering if you can expand on how that hangs together? Thanks! Rob On 31 October 2013 14:01, Adrian Otto adrian.o...@rackspace.com wrote: Clayton, Thanks for adding the diagram illustrating the flow. I expect that respond to client may not be a synchronous flow, rather that if creation of a repo takes a while that the API may return a 202 Accepted, and the client can poll a status attribute to determine completion and learn the actual location of the repo created. So in that case respond to client also might mean update async status. This will be particularly important in cases where an external SCM system is used (perhaps Github). I agree that this will be a helpful reference to guide our upcoming interactive design sessions. I have created the first call for participation, which will also be separately announced: https://wiki.openstack.org/wiki/Solum/BreakoutMeetings Adrian On Oct 30, 2013, at 2:45 PM, Clayton Coleman ccole...@redhat.com wrote: In the IRC meeting yesterday [1] we discussed splitting the individual topics for the git deploy blueprint [2] into rough sub areas. To help frame the abstractions we've been discussing I roughed out a flow based on two user inputs, REST API create and a git push and then tried to draw boxes around the related bits. The abstractions roughly correspond to the potential areas that can be the focus of break out discussions (and if they don't a good discussion of why is always helpful). Squiggly boxes are potential plugin
Re: [openstack-dev] [Solum] Separation of concerns for 0.1 git deploy blueprint
Hi Clayton, Thank you for creating these diagrams. They are a great starting point for discussions around the git deploy blueprint. Some questions/comments: In both the diagrams, what do the arrows indicate? Data flow, control-flow, or some kind of ordering relationship? Especially, the arrow from 'respond to client' to 'HEAT' box in the top diagram is a bit confusing. Correct me if I am wrong, but my guess is that the meaning of that arrow is, 'stack create' will happen after responding to client has happened. Although, this may not be necessarily true (both those steps can happen in async manner). Second, how do you envision tying together the two diagrams/high-level flows? Specifically, once we have a built artifact (second diagram), how do we go about deploying it? We could either have, build abstraction - REST API - deploy abstraction, or, build abstraction - deploy abstraction. What are your thoughts on this? Best regards, Devdatta -Original Message- From: Adrian Otto adrian.o...@rackspace.com Sent: Wednesday, October 30, 2013 8:01pm To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.org Subject: Re: [openstack-dev] [Solum] Separation of concerns for 0.1 git deploy blueprint Clayton, Thanks for adding the diagram illustrating the flow. I expect that respond to client may not be a synchronous flow, rather that if creation of a repo takes a while that the API may return a 202 Accepted, and the client can poll a status attribute to determine completion and learn the actual location of the repo created. So in that case respond to client also might mean update async status. This will be particularly important in cases where an external SCM system is used (perhaps Github). I agree that this will be a helpful reference to guide our upcoming interactive design sessions. I have created the first call for participation, which will also be separately announced: https://wiki.openstack.org/wiki/Solum/BreakoutMeetings Adrian On Oct 30, 2013, at 2:45 PM, Clayton Coleman ccole...@redhat.com wrote: In the IRC meeting yesterday [1] we discussed splitting the individual topics for the git deploy blueprint [2] into rough sub areas. To help frame the abstractions we've been discussing I roughed out a flow based on two user inputs, REST API create and a git push and then tried to draw boxes around the related bits. The abstractions roughly correspond to the potential areas that can be the focus of break out discussions (and if they don't a good discussion of why is always helpful). Squiggly boxes are potential plugin points and/or strong responsibility boundaries. Feedback desirable. [1] http://irclogs.solum.io/2013/solum.2013-10-29-16.01.txt [2] https://wiki.openstack.org/wiki/Solum/FeatureBlueprints/GitIntegration#Flow_Boundaries ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
