Hi all, I've recently started looking at the various methods for deploying and developing tripleo. What I would like to bring up is the current combination of the tooling for managing the VM instances and the actual deployment method to launch the undercloud/overcloud installation. While running through the various methods and reading up on the documentation, I'm concerned that they are not currently flexible enough for a developer (or operator for that matter) to be able to setup the various environment configurations for testing deployments and doing development. Additionally I ran into issues just trying get them working at all so this probably doesn't help when trying to attract new contributors as well. The focus of this email and of my experience seems to relate with workflow-simplification spec[0]. I would like to share my experiences with the various tooling available and raise some ideas.
Example Situation: For example, I have a laptop with 16G of RAM and an SSD and I'd like to get started with tripleo. How can I deploy tripleo? Tools: instack: I started with the tripleo docs[1] that reference using the instack tools for virtual environment creation while deploying tripleo. The docs say you need at least 12G of RAM[2]. The docs lie (step 7[3]). So after basically shutting everything down and letting it deploy with all my RAM, the deployment fails because the undercloud runs out of RAM and OOM killer kills off heat. This was not because I had reduced the amount of ram for the undercloud node or anything. It was because by default, 6GB of RAM with no swap is configured for the undercloud (not sure if this is a bug?). So I added a swap file to the undercloud and continued. My next adventure was having the overcloud deployment fail because lack of memory as puppet fails trying to spawn a process and gets denied. The instack method does not configure swap for the VMs that are deployed and the deployment did not work with 5GB RAM for each node. So for a full 16GB I was unable to follow the documentation and use instack to successfully deploy. At this point I switched over to trying to use tripleo-quickstart. Eventually I was able to figure out a configuration with instack to get it to deploy when I figured out how to enable swap for the overcloud deployment. tripleo-quickstart: The next thing I attempted to use was the tripleo-quickstart[4]. Following the directions I attempted to deploy against my localhost. I turns out that doesn't work as expected since ansible likes to do magic when dealing with localhost[5]. Ultimately I was unable to get it working against my laptop locally because I ran into some libvirt issues. But I was able to get it to work when I pointed it at a separate machine. It should be noted that tripleo-quickstart creates an undercloud with swap which was nice because then it actually works, but is an inconsistent experience depending on which tool you used for your deployment. Thoughts: What these two methods showed me is that the deployment of tripleo is not exactly a foolproof thing and that there are a lot of assumptions that are being handled by the both of these tools. My initial goal to start this conversation around tooling and workflows was to bring the idea of separation of the (virtual) environment configuration from the actual deployment of tripleo as well as identifying places for improvement as a way to speed up development and deployment testing. I believe there are a few reasons why this can be beneficial. The first reason is that as a developer, I would like to simplify the development environment creation process and be able to draw the line between environment and actual deployment tool. By developing and documenting a working development/deployment workflow, we can simplify the onboarding experience as well as possibly accelerating the existing development processes by reducing the time spent messing with creating environments. Does tripleo need to manage creation of VMs to deploy on? The answer is probably no. As the end user will want to deploy tripleo on his or her gear, the focus for tripleo probably should be on improving that process. Now this doesn't mean that we can't write stuff to do this, as it's important for development and testing. I'm not sure this is a core part of what should be 'tripleo'. Another reason why I think this is important is as we talk about creating different scenarios for CI[6] to improve testing, it would also be useful for a developer or qa engineer to be able to test different environmental configurations that would be more realistic of actual deployment scenarios without having to hunt down multiple machines or configure actual hardware networking. For example, creating environments using multiple networks, changing NICs, providing different sized nodes based on roles, etc can all be done virtually. While tripleo-quickstart has some of these options, it is mixed in with the tripleo deployment process and does not seem to align with being able to deploy tripleo in more real world networking or environmental scenarios. Since there are a bunch of assumptions baked into the existing development scripts, I would say the current approach is more 'it works in devstack' than 'it works for the end user'. This is not to say the currently tools don't have their uses as they currently work for the existing CI setup and for many developers today. I think we can do better if we draw clearer lines between what is tripleo and what is something that is environmental and a supporting tool. Ideas: As part of bringing something to get the conversation started and to better understand how things work, I spent about two days coming up with a PoC[7] for a workflow that splits the environment creation, configuration, and management out from the actual deployment of the undercloud/overcloud. Having previously used other tools for managing environments for deploying openstack, I thought I'd try deploying tripleo using something I was familiar with, fuel-devops[8]. The whole point of fuel-devops is to be able to create and manage entire virtual environments (and their networking configuration) on a given host. With this, I was able to create my environment setup in a yaml file[9] which would then be able to be reproduced. So with this tool, I'm able to create a number nodes of a given memory, disk, network configuration as part of an 'environment'. This environment is completely separated from another environment which means given a large enough virtual host, I could have multiple tripleo deployments occurring simultaneously on their own networks. This is a nice feature, but just an added bonus to the tool (along with snapshotting and a few other nifty things). The bigger feature here is that this is more representative of what someone using tripleo is going to experience. They are going to have their environment already configured and would like to deploy tripleo on it. Once the environment was created, I started to understand what it would be like for an end user to take an undercloud image and deploy it. Fortunately because we're still dealing with VMs, you can just point the undercloud node at the undercloud image itself[10] for testing purposes. Once the environment exists, it starts exposing what exactly it means to deploy a tripleo undercloud/overcloud. The majority of the effort I had to expend for this PoC was actually related to the construction of the instackenv.json to load the overcloud nodes into ironic. As mentioned in the workflow-simplification spec[0], this is a known limitation and there are possible solutions and I think this is important of the end user experience when trying to work with tripleo. It should be noted that I managed to get the undercloud and controller/compute deployed (but eating into VM swap space) in 12GB on my laptop. This was something I was unable to do with either instack or tripleo-quickstart. There are some short coming with this particular tool choice. My understanding is that fuel-devops is still limited to managing a single host. So you don't use it against remote nodes, but it is good if you have decently sized physical machine or want to work locally. I ran into issues with network configurations and pxe booting, but I have a feeling that's more of a bug in libvirt and my lack of time to devote to undercloud setup. So it's not perfect, but it does show off the basics of the concept. Overall I think clearly defining the tripleo installation process from the environment configuration is an important step for end user usability and even developer workflows. Thanks, -Alex [0] https://review.openstack.org/238192 [1] http://docs.openstack.org/developer/tripleo-docs/environments/environments.html#virtual-environment [2] http://docs.openstack.org/developer/tripleo-docs/environments/environments.html#minimum-system-requirements [3] http://docs.openstack.org/developer/tripleo-docs/environments/environments.html#preparing-the-virtual-environment-automated [4] https://github.com/openstack/tripleo-quickstart [5] http://eavesdrop.openstack.org/irclogs/%23rdo/%23rdo.2016-09-06.log.html#t2016-09-06T19:33:44 [6] http://lists.openstack.org/pipermail/openstack-dev/2016-August/101786.html [7] https://github.com/mwhahaha/fuel-devops-tripleo [8] https://github.com/openstack/fuel-devops [9] https://github.com/mwhahaha/fuel-devops-tripleo/blob/master/ooo-template.yaml [10] https://github.com/mwhahaha/fuel-devops-tripleo/blob/master/ooo-template.yaml#L78 __________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev