Apologies for top post, but I just wanted to point out in the config-external examples I am aware /opt/kolla is the wrong directory to configure from (it should be /var/lib/kolla or something similar) and we will fix this during l3.
Regards -steve On 8/5/15, 6:53 PM, "Steven Dake (stdake)" <[email protected]> wrote: > > >On 8/5/15, 11:33 AM, "Ryan Hallisey" <[email protected]> wrote: > >>Tagging kolla so the kolla community also sees it. >>Pardon the top posting. >> >>-Ryan >Ryan, > >Super appreciated I may have missed this in my deluge of mail - please >keep doing that - or if it involves kolla please tag as kolla as I read >that folder often :) > >Dan, > >Thanks for the detailed posting. > >Comments inline. >> >>----- Original Message ----- >>From: "Dan Prince" <[email protected]> >>To: "openstack-dev" <[email protected]> >>Sent: Wednesday, August 5, 2015 2:29:13 PM >>Subject: [openstack-dev] [TripleO] [Puppet] Deploying OpenStack with >>Puppet modules on Docker with Heat >> >>Hi, >> >>There is a lot of interest in getting support for container based >>deployment within TripleO and many different ideas and opinions on how >>to go about doing that. >> >>One idea on the table is to use Heat to help orchestrate the deployment >>of docker containers. This would work similar to our tripleo-heat >>-templates implementation except that when using docker you would swap >>in a nested stack template that would configure containers on >>baremetal. We've even got a nice example that shows what a >>containerized TripleO overcloud might look like here [1]. The approach >>outlines how you might use kolla docker containers alongside of the >>tripleo-heat-templates to do this sort of deployment. >> >>This is all cool stuff but one area of concern is how we do the actual >>configuration of the containers. The above implementation relies on >>passing environment variables into kolla built docker containers which >>then self configure all the required config files and start the >>service. This sounds like a start... but creating (and maintaining) >>another from scratch OpenStack configuration tool isn't high on my list > >Agree we came to the exact same conclusion which is why we changed to the >config-external config strategy in the ansible-multi spec at: > >https://review.openstack.org/#/c/189157/ > > >>of things to spend time on. Sure there is already a kolla community >>helping to build and maintain this configuration tooling (mostly >>thinking config files here) but this sounds a bit like what tripleo >>-image-elements initially tried to do and it turns out there are much >>more capable configuration tools out there. > >We would prefer to drop config-internal if TripleO doesn¹t plan to use it >and focus on config-external. The reason we left it intact is we didn¹t >want to leave TripleO in the lurch. > >> >>Since we are already using a good bit of Puppet in tripleo-heat >>-templates the idea came up that we would try to configure Docker >>containers using Puppet. Again, here there are several ideas in the >>Puppet community with regards to how docker might best be configured >>with Puppet. Keeping those in mind we've been throwing some ideas out >>on an etherpad here [2] that describes using Heat for orchestration, >>Puppet for configuration, and Kolla docker images for containers. >> >>A quick outline of the approach is: >> >>-Extend the heat-container-agent [3] that runs os-collect-config and >>all the required hooks we require for deployment. This includes docker >>-compute, bash scripts, and Puppet. NOTE: As described in the etherpad >>I've taken to using DIB to build this container. I found this to be >>faster from a TripleO development baseline. >> >>-To create config files the heat-container-agent would run a puppet >>manifest for a given role and generate a directory tree of config files >>(/var/lib/etc-data for example). > >Would prefer per file mounting for config-external. (see more below) > >> >>-We then run a docker-compose software deployment that mounts those >>configuration file(s) into a read only volume and uses them to start >>the containerized service. > >Sounds good. > >> >>The approach could look something like this [4]. This nice thing about >>this is that it requires no modification to OpenStack Puppet modules. >>We can use those today, as-is. Additionally, although Puppet runs in >>the agent container we've created a mechanism to set all the resources >>to noop mode except for those that generate config files. And lastly, >>we can use exactly the same role manifest for docker that we do for >>baremetal. Lots of re-use here... and although we are disabling a lot >>of Puppet functionality in setting all the non-config resources to noop >>the Kolla containers already do some of that stuff for us (starting >>services, etc.). >> >>---- >> >>All that said (and trying to keep this short) we've still got a bit of >>work to do around wiring up externally created config files to kolla >>build docker containers. A couple of issues are: >> >>-The external config file mechanism for Kolla containers only seems to >>support a single config file. Some services (Neutron) can have multiple >>files. Could we extend the external config support to use multiple >>files? > >Kolla supports multiple configuration files. An example: > >https://github.com/stackforge/kolla/blob/master/docker/common/neutron/neut >r >on-server/config-external.sh > > >We will extend to new config files as it makes sense and I don¹t really >feel all that terrible about importing even proprietary config file >formats for specific proprietary drivers that neutron (or cinder or >whatever) may need if there was an actual interest and the file format >were published upstream somewhere so we could process it properly in >Ansible if we choose to do so at a later time. > >> >>-If a service has multiple files kolla may need to adjust its service >>startup script to use multiple files. Perhaps a conf.d approach would >>work here? > >A conf.d approach could work, but I¹d prefer to use the design pattern in >Neutron above. The downside of the neutron pattern is each file must be >separately specified. The reason I¹d prefer this approach is it maintains >container immutability without pollution of files into the container. >With a directory, new files could be loaded into the etc directory without >directly restarting the container with a :ro flag. This would break the >immutability of the container, and I¹m not sure how to protect against >that happening with a directory import. This is why we import only files. > >> >>-We are missing published version of some key kolla containers. Namely >>openvswitch and the neutron-openvswitch-agent for starters but I'd also >>like to have a Ceilometer agent and SNMP agent container as well so we >>have feature parity with the non-docker compute role. > >Openvswitch works fantastically today and was our first Ansible deployment >mechanism. We later added linuxbridge support and plan nova-networking >for l3. We don¹t plan to port this work to config-internal config >strategy, but as stated in our ansible-multi review, we won¹t turn away >patches that implement this work in the config-internal model. > >Our ovs work is here: > >https://github.com/stackforge/kolla/tree/master/docker/common/openvswitch >https://github.com/stackforge/kolla/tree/master/docker/common/neutron/neut >r >on-openvswitch-agent > > >Ceilometer agent - agreed this is needed and high on my list for l3. Not >familiar with an SNMP agent but it doesn¹t sound too difficult to take on. > If you can provide more details on what to configure, I¹ll get a >blueprint filed and we can get this work prioritized for l3. Our >ceilometer container in general needs some love to be tackled in l3. > >It is likely you looked at the repo a couple weeks ago when this stuff >wasn¹t fully baked. I realize we are moving pretty fast so no harm on the >lag :) > >Regards >-steve > >> >>Once we have solutions for the above I think we'll be very close to a >>fully dockerized compute role with TripleO heat templates. From there >>we can expand the idea to cover other roles within the tripleo-heat >>-templates too. >> >>I'll stop there for now. Any ideas and thoughts appreciated. >> >>Dan >> >>----- >> >>[1] https://review.openstack.org/#/c/178840/ (Containerized TripleO >>Overcloud.) >>[2] https://etherpad.openstack.org/p/tripleo-docker-puppet >>[3] http://git.openstack.org/cgit/openstack/heat >>-templates/log/hot/software-config/heat-container-agent >>[4] https://review.openstack.org/#/c/209505/ (Docker compute role >>configured via Puppet) >> >>_________________________________________________________________________ >>_ >>OpenStack Development Mailing List (not for usage questions) >>Unsubscribe: >>[email protected]?subject:unsubscribe >>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> >>_________________________________________________________________________ >>_ >>OpenStack Development Mailing List (not for usage questions) >>Unsubscribe: >>[email protected]?subject:unsubscribe >>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > __________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: [email protected]?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
