Re: [openstack-dev] [tripleo] Patches to speed up plan operations
Hey Jirka! I wasn't aware of the other options available. Basically yes, you now just need to upload a tarball of the templates to swift. You can see in the client: -tarball.tarball_extract_to_swift_container( -swift_client, tmp_tarball.name, container_name) +_upload_file(swift_client, container_name, + constants.TEMPLATES_TARBALL_NAME, tmp_tarball.name) Other than that it should be the same. I'm not sure what files the UI wants to look at in swift, but certainly some are still there. Basically any file that exists in swift is not overwritten by the contents of the tar file. So if a file exists in swift it takes priority. I'll try to catch you on irc but I know our timezones are quite different. Thanks for looking into it! Ian On Wed, Aug 8, 2018 at 10:46 AM Jiri Tomasek wrote: > Hello, thanks for bringing this up. > > I am going to try to test this patch with TripleO UI tomorrow. Without > properly looking at the patch, questions I would like to get answers for > are: > > How is this going to affect ways to create/update deployment plan? > Currently user is able to create deployment plan by: > - not providing any files - creating deployment plan from default files in > /usr/share/openstack-tripleo-heat-templates > - providing a tarball > - providing a local directory of files to create plan from > - providing a git repository link > > These changes will have an impact on certain TripleO UI operations where > (in rare cases) we reach directly for a swift object > > IIUC it seems we are deciding to consider deployment plan as a black box > packed in a tarball, which I quite like, we'll need to provide a standard > way how to provide custom files to the plan. > > How is this going to affect CLI vs GUI workflow as currently CLI creates > the plan as part of the deploy command, rather than GUI starts its workflow > by selecting/creating deployment plan and whole configuration of the plan > is performed on the deployment plan. Then the deployment plan gets > deployed. We are aiming to introduce CLI commands to consolidate the > behaviour of both clients to what GUI workflow is currently. > > I am going to try to find answers to these questions and identify > potential problems in next couple of days. > > -- Jirka > > > On Tue, Aug 7, 2018 at 5:34 PM Dan Prince wrote: > >> Thanks for taking this on Ian! I'm fully on board with the effort. I >> like the consolidation and performance improvements. Storing t-h-t >> templates in Swift worked okay 3-4 years ago. Now that we have more >> templates, many of which need .j2 rendering the storage there has >> become quite a bottleneck. >> >> Additionally, since we'd be sending commands to Heat via local >> filesystem template storage we could consider using softlinks again >> within t-h-t which should help with refactoring and deprecation >> efforts. >> >> Dan >> On Wed, Aug 1, 2018 at 7:35 PM Ian Main wrote: >> > >> > >> > Hey folks! >> > >> > So I've been working on some patches to speed up plan operations in >> TripleO. This was originally driven by the UI needing to be able to >> perform a 'plan upload' in something less than several minutes. :) >> > >> > https://review.openstack.org/#/c/581153/ >> > https://review.openstack.org/#/c/581141/ >> > >> > I have a functioning set of patches, and it actually cuts over 2 >> minutes off the overcloud deployment time. >> > >> > Without patch: >> > + openstack overcloud plan create --templates >> /home/stack/tripleo-heat-templates/ overcloud >> > Creating Swift container to store the plan >> > Creating plan from template files in: >> /home/stack/tripleo-heat-templates/ >> > Plan created. >> > real3m3.415s >> > >> > With patch: >> > + openstack overcloud plan create --templates >> /home/stack/tripleo-heat-templates/ overcloud >> > Creating Swift container to store the plan >> > Creating plan from template files in: >> /home/stack/tripleo-heat-templates/ >> > Plan created. >> > real0m44.694s >> > >> > This is on VMs. On real hardware it now takes something like 15-20 >> seconds to do the plan upload which is much more manageable from the UI >> standpoint. >> > >> > Some things about what this patch does: >> > >> > - It makes use of process-templates.py (written for the undercloud) to >> process the jinjafied templates. This reduces replication with the >> existing version in the code base and is very fast as it's all done on >&
[openstack-dev] [tripleo] Patches to speed up plan operations
Hey folks! So I've been working on some patches to speed up plan operations in TripleO. This was originally driven by the UI needing to be able to perform a 'plan upload' in something less than several minutes. :) https://review.openstack.org/#/c/581153/ https://review.openstack.org/#/c/581141/ I have a functioning set of patches, and it actually cuts over 2 minutes off the overcloud deployment time. Without patch: + openstack overcloud plan create --templates /home/stack/tripleo-heat-templates/ overcloud Creating Swift container to store the plan Creating plan from template files in: /home/stack/tripleo-heat-templates/ Plan created. real3m3.415s With patch: + openstack overcloud plan create --templates /home/stack/tripleo-heat-templates/ overcloud Creating Swift container to store the plan Creating plan from template files in: /home/stack/tripleo-heat-templates/ Plan created. real0m44.694s This is on VMs. On real hardware it now takes something like 15-20 seconds to do the plan upload which is much more manageable from the UI standpoint. Some things about what this patch does: - It makes use of process-templates.py (written for the undercloud) to process the jinjafied templates. This reduces replication with the existing version in the code base and is very fast as it's all done on local disk. - It stores the bulk of the templates as a tarball in swift. Any individual files in swift take precedence over the contents of the tarball so it should be backwards compatible. This is a great speed up as we're not accessing a lot of individual files in swift. There's still some work to do; cleaning up and fixing the unit tests, testing upgrades etc. I just wanted to get some feedback on the general idea and hopefully some reviews and/or help - especially with the unit test stuff. Thanks everyone! Ian __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tripleo] [ci]
Wesley Hayutin wrote: > Greetings, > > I wanted to send a status update on the quickstart based containerized > compute ci. > > The work is here: > https://review.openstack.org/#/c/393348/ > > I had two passes on the morning of Nov 30 in a row, then later that day the > deployment started to fail due the compute node loosing it's networking and > became unpingable. After poking around and talking to a few folks its > likely that we're hitting at least one of two possible bugs [1-2] > > I am on pto next week but will periodically check in and can easily retest > if these resolve. I've been seeing this a lot too. It's happening to both the controller and compute for me. Probably because the controller is ALSO running the firstboot script in docker/ which is not what we want (or we need it to be smarter anyway). So far it appears that cloud-init is running our firstboot script but is not configuring networking. If I run dhclient eth0 it comes up and has internet access etc. Going to look into this more tomorrow. Ian __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [TripleO][Heat][Kolla][Magnum] The zen of Heat, containers, and the future of TripleO
Fox, Kevin M wrote: > +1 for TripleO taking a look at Kolla. > > Some random thoughts: > > I'm in the middle of deploying a new cloud and I couldn't use either TripleO > or Kolla for various reasons. A few reasons for each: > * TripeO - worries me for ever having to do a major upgrade of the software, > or needing to do oddball configs like vxlans over ipoib. > * Kolla - At the time it was still immature. No stable artefacts posted. > database container recently broke, little documentation for disaster > recovery. No upgrade strategy at the time. > > Kolla rearchitected recently to support oddball configs like we've had to do > at times. They also recently gained upgrade support. I think they are on the > right path. If I had to start fresh, I'd very seriously consider using it. > > I think Kolla can provide the missing pieces that TripleO needs. TripleO has > bare metal deployment down solid. I really like the idea of using OpenStack > to deploy OpenStack. Kolla is now OpenStack so should be considered. > > I'm also in favor of using Magnum to deploy a COE to manage Kolla. I'm much > less thrilled about Mesos though. It feels heavy enough weight that it feels > like your deploying an OpenStack like system just to deploy OpenStack. So, > OpenStack On NotOpenStack On OpenStack. :/ I've had good luck with Kubernetes > (much simpler) recently and am disappointed that it was too immature at the > time Kolla originally considered it. It seems much more feasible to use now. > I use net=host like features all the time which was a major sticking point > before. > > I'd be interested in seeing TripeO use the Ansible version for now since > that's working, stable, and supports upgrades/oddball configs. Then in the > future as Kubernetes support or maybe Mesos support matures, consider that. > Kolla's going to have to have a migration path from one to the other > eventually... I think this would allow TripeO to really come into its own as > an end to end, production ready system sooner. > > Thanks, > Kevin This is very much my thinking as well. I like your pragmatic take on it. The community is building solutions to these problems and we should join them. Ian __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova][docker][containers][qa] nova-docker CI failing a lot on unrelated nova patches
Sean Dague wrote: On 12/04/2014 05:38 PM, Matt Riedemann wrote: On 12/4/2014 4:06 PM, Michael Still wrote: +Eric and Ian On Fri, Dec 5, 2014 at 8:31 AM, Matt Riedemann mrie...@linux.vnet.ibm.com wrote: This came up in the nova meeting today, I've opened a bug [1] for it. Since this isn't maintained by infra we don't have log indexing so I can't use logstash to see how pervasive it us, but multiple people are reporting the same thing in IRC. Who is maintaining the nova-docker CI and can look at this? It also looks like the log format for the nova-docker CI is a bit weird, can that be cleaned up to be more consistent with other CI log results? [1] https://bugs.launchpad.net/nova-docker/+bug/1399443 -- Thanks, Matt Riedemann ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev Also, according to the 3rd party CI requirements [1] I should see nova-docker CI in the third party wiki page [2] so I can get details on who to contact when this fails but that's not done. [1] http://ci.openstack.org/third_party.html#requirements [2] https://wiki.openstack.org/wiki/ThirdPartySystems It's not the 3rd party CI job we are talking about, it's the one in the check queue which is run by infra. But, more importantly, jobs in those queues need shepards that will fix them. Otherwise they will get deleted. Clarkb provided the fix for the log structure right now - https://review.openstack.org/#/c/139237/1 so at least it will look vaguely sane on failures -Sean This is one of the reasons we might like to have this in nova core. Otherwise we will just keep addressing issues as they come up. We would likely be involved doing this if it were part of nova core anyway. Ian -- Sean Dague http://dague.net ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [kolla] on Dockerfile patterns
Angus Lees wrote: I've been reading a bunch of the existing Dockerfiles, and I have two humble requests: 1. It would be good if the interesting code came from python sdist/bdists rather than rpms. This will make it possible to rebuild the containers using code from a private branch or even unsubmitted code, without having to go through a redhat/rpm release process first. I care much less about where the python dependencies come from. Pulling them from rpms rather than pip/pypi seems like a very good idea, given the relative difficulty of caching pypi content and we also pull in the required C, etc libraries for free. With this in place, I think I could drop my own containers and switch to reusing kolla's for building virtual testing environments. This would make me happy. 2. I think we should separate out run the server from do once-off setup. Currently the containers run a start.sh that typically sets up the database, runs the servers, creates keystone users and sets up the keystone catalog. In something like k8s, the container will almost certainly be run multiple times in parallel and restarted numerous times, so all those other steps go against the service-oriented k8s ideal and are at-best wasted. I suggest making the container contain the deployed code and offer a few thin scripts/commands for entrypoints. The main replicationController/pod _just_ starts the server, and then we have separate pods (or perhaps even non-k8s container invocations) that do initial database setup/migrate, and post- install keystone setup. I'm open to whether we want to make these as lightweight/independent as possible (every daemon in an individual container), or limit it to one per project (eg: run nova-api, nova-conductor, nova-scheduler, etc all in one container). I think the differences are run-time scalability and resource- attribution vs upfront coding effort and are not hugely significant either way. Post-install catalog setup we can combine into one cross-service setup like tripleO does[1]. Although k8s doesn't have explicit support for batch tasks currently, I'm doing the pre-install setup in restartPolicy: onFailure pods currently and it seems to work quite well[2]. (I'm saying post install catalog setup, but really keystone catalog can happen at any point pre/post aiui.) [1] https://github.com/openstack/tripleo-incubator/blob/master/scripts/setup-endpoints [2] https://github.com/anguslees/kube-openstack/blob/master/kubernetes-in/nova-db-sync-pod.yaml -- - Gus One thing I've learned is to not perform software updates within a container. A number of the containers I've seen do software updates on startup but I've seen this break dependencies in containers a few times making them unusable. This detracts from the ability to have a completely controlled environment within a container with proven software versions that play nicely together. Ian ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
