Uri, Yes. We will need to offer a facility for customizing the CD pipeline(s). We have not contemplated all of the possibilities for that yet, so this is something that we will certainly discuss further. We will need to strike the right balance between options and simplicity.
We have thought through the possibility of generating a HOT artifact in the case that one is not provided, and allowing a user-supplied one in the event that one is supplied with the code at the time of the deployment action. Of course if no HOT is provided, we can generate one (or a simple hierarchy of them). That option would allow you to cause anything that Heat knows how to do. We will continue to contribute to Heat (and other downstream projects) to help it achieve its mission. From my view, Heat should allow us to connect various different imperative systems downstream. For example, if you want something to happen on a given node type, you might use Chef or Puppet to accomplish that. Features related to orchestration (the things offered in the HOT as declarations) should be provided by heat, or one of its resource types. That's where I see most features like rollback fitting in. The actual design of the logic that allows the promotion of a deployment through different Environments will likely be triggered by Solum, but implemented downstream in Heat. A system level rollback (user requested, rather than initiated by a failure) may actually be achieved by a request to swap Environments, or modify one. Adrian References: : https://wiki.openstack.org/wiki/Heat/DSL#Heat_Orchestration_Template_.28HOT.29 : https://wiki.openstack.org/wiki/Heat/DSL#Environment On Oct 31, 2013, at 2:04 AM, Uri Cohen <u...@gigaspaces.com> wrote: > Hi Adrian > > Sounds like there's a need for a true workflow engine here. I suspect that > for any use case beyond trivial, each user will have her own flavor of CD > (TripleO is a good example for this) - some will prefer to use a canary > server and test it manually before deploying to other servers, some will > deploy to specific servers based on the subset of users they're serving, some > will do a red/black type of deployment (Which maps well into environments > below), etc. But I think that there should be an user-configurable mechanism > to define and tweak these workflows. There will also be additional related > workflows - e.g. rolling back a deployment, or updating the infrastructure > components of a running application (e.g. database version upgrade). > I might be raising a topic that's already been discussed (sorry if I am), but > I was wondering how this would fit into the architecture. > > Thanks! > Uri > > On Oct 31, 2013, at 6:42 AM, Adrian Otto <adrian.o...@rackspace.com> wrote: > >> Robert, >> >> We would be interested in exploring your use case for CD, and help you judge >> what would be the best fit. I know that Monty Taylor has identified some >> parts of openstack-infra that we could potentially leverage in Solum to >> address our CI/CD goals. The design of that feature is still pending, so we >> are looking forward to sourcing its of input on that. >> >> I don't anticipate any fundament gaps that would prevent you from using a >> future version of Solum to help narrow your scope. The whole point of our >> focus is to try to make these goals easier to achieve, and try to get >> maximum leverage of what the OpenStack ecosystem already offers. >> >> The high level concept behind the GIt Integration feature currently >> conceived for Solum is to offer application developers a way to commit code >> to a Git repo and upon a git push command, have a remote system trigger a >> modular CI/CD pipeline that ends with that application running on an >> OpenStack cloud in one or more "environments" (dev/test/stage/prod/etc.). In >> a simple case that may be (in the case of a dynamic language like Python) >> the testing of the code, arrangement of the commit into a deployable bundle, >> triggering of the Solum API to trigger a deployment, automatic generation of >> a HOT artifact, and stimulation of the Heat API and any other API calls >> needed. >> >> In a more complex scenario, you may want additional features and >> capabilities, such as a system like Gerrit to trigger the commit/push, >> integrate an artifact repository for storing the builds, etc. Using Zuul for >> that would be appropriate for sure. >> >> There are also dev shops who already have a lot invested into their Jenkins >> setups, and will want to bypass portions of what I described above, so there >> will be a use case supported where you can just integrate with the solum >> API, and pass in a pre-built bundle artifact and bypass part or all of >> Solum's CI features, and just use it for deployment and re-deployment and >> promotion of a deployment between various environments (dev/test/pros/etc.). >> You would expect features like canary deploy in that case that might make it >> more attractive than just generating your own HOT artifact in your Jenkins >> workflow and passing that in directly for orchestration. >> >> You should expect a higher level of control for management of the deployed >> system with abstracts that let you easily extend the system for monitoring >> and management reasons. >> >> Note that not all this functionality belongs in Solum. Some of it may >> actually come from Heat, and other downstream systems. The idea is that we >> address the key gaps. >> >> Adrian >> >> On Oct 30, 2013, at 9:16 PM, Robert Collins <robe...@robertcollins.net> >> wrote: >> >>> Hi Adrian, >>> I'm very interested in Solum, particularly as it relates to >>> TripleO : as you know the TripleO story is to treat OpenStack as just >>> another application to install via Heat/Nova etc - and we've got a >>> major story around pulling from git -> CI + CD pipeline -> automated >>> deployment to production. >>> >>> So - where we can say 'hey, this should be part of Solum', we can hive >>> stuff out of the TripleO program into Solum - but only if we've got >>> compatible stories. >>> >>> Right now, our design for CD is basically Gerrit + Zuul - exactly the >>> same as gating checks for OpenStack itself. When I look at >>> https://wiki.openstack.org/wiki/Solum/FeatureBlueprints/GitIntegration#Flow_Boundaries >>> I'm having a little trouble mapping that into your design. >>> >>> I'm wondering if you can expand on how that hangs together? >>> >>> Thanks! >>> Rob >>> >>> On 31 October 2013 14:01, Adrian Otto <adrian.o...@rackspace.com> wrote: >>>> Clayton, >>>> >>>> Thanks for adding the diagram illustrating the flow. I expect that >>>> "respond to client" may not be a synchronous flow, rather that if creation >>>> of a repo takes a while that the API may return a 202 Accepted, and the >>>> client can poll a status attribute to determine completion and learn the >>>> actual location of the repo created. So in that case "respond to client" >>>> also might mean "update async status". This will be particularly important >>>> in cases where an external SCM system is used (perhaps Github). >>>> >>>> I agree that this will be a helpful reference to guide our upcoming >>>> interactive design sessions. I have created the first call for >>>> participation, which will also be separately announced: >>>> >>>> https://wiki.openstack.org/wiki/Solum/BreakoutMeetings >>>> >>>> Adrian >>>> >>>> On Oct 30, 2013, at 2:45 PM, Clayton Coleman <ccole...@redhat.com> wrote: >>>> >>>>> In the IRC meeting yesterday  we discussed splitting the individual >>>>> topics for the git deploy blueprint  into rough sub areas. To help >>>>> frame the abstractions we've been discussing I roughed out a flow based >>>>> on two user inputs, REST API create and a git push and then tried to draw >>>>> boxes around the related bits. The abstractions roughly correspond to >>>>> the potential areas that can be the focus of break out discussions (and >>>>> if they don't a good discussion of why is always helpful). >>>>> >>>>> Squiggly boxes are potential plugin points and/or strong responsibility >>>>> boundaries. Feedback desirable. >>>>> >>>>>  http://irclogs.solum.io/2013/solum.2013-10-29-16.01.txt >>>>>  >>>>> https://wiki.openstack.org/wiki/Solum/FeatureBlueprints/GitIntegration#Flow_Boundaries >>>>> >>>>> >>>>> >>>>> _______________________________________________ >>>>> OpenStack-dev mailing list >>>>> OpenStackemail@example.com >>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>>> >>>> >>>> _______________________________________________ >>>> OpenStack-dev mailing list >>>> OpenStackfirstname.lastname@example.org >>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>> >>> >>> >>> -- >>> Robert Collins <rbtcoll...@hp.com> >>> Distinguished Technologist >>> HP Converged Cloud >>> >>> _______________________________________________ >>> OpenStack-dev mailing list >>> OpenStackemail@example.com >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> >> >> _______________________________________________ >> OpenStack-dev mailing list >> OpenStackfirstname.lastname@example.org >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > _______________________________________________ > OpenStack-dev mailing list > OpenStackemail@example.com > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev _______________________________________________ OpenStack-dev mailing list OpenStackfirstname.lastname@example.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev