Re: [openstack-dev] [OpenStack-Infra] There is no Jenkins, only Zuul
Are 3d party CIs affected by this change and if yes how? What's the recommendation for newly created 3rd party CIs - should they use new zuul only or it's fine to use existing (and tested) jenkins+zuul configuration? Second question - if jobs are to be launched by zuul+ansible, how will the worker host be chosen, by ansible-launcher? (normally as I understand Jenkins master takes care of deciding which hosts are free to run jobs) Are there any specs about way of managing workers (adding/removing, setting # of executors)? Thank you. -marek -- Marek Zawadzki Mirantis Containerized Control Plane Team __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [kolla][infra] Publishing kolla images to docker-registry.openstack.org
ect:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Marek Zawadzki Mirantis Kolla Team __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [kolla][nova] Orchiestrated upgrades in kolla
Thank you for clarification. -marek On 17.12.2015 18:02, Michał Jastrzębski wrote: Hey, Since our instances (and qemu) is in different container than nova services, there is no reason (specific to kolla) to meddle with them. Ofc it would be required if you'd like to upgrade qemu, but that's beyond scope of this change. For now we assume that project upgrade won't affect running vms, or if that is the case, operator will have to migrate it himself (or herself). Problem is with orchiestraded ansible it will require a bit more tinkering to run playbook on just one node (namely one we migrated from), but that's doable. Our default upgrade strategy assumes that there is no downtime causing change. Michal __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Marek Zawadzki Mirantis Kolla Team __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [kolla] Upgrading containers in Mesos/Marathon environment - request for feedback
Update - you can take a look into comments about Mesos maintaining upgrade process + my response to it that breaks the topic into a list of OpenStack services (starting with a minimal list that is necessary to build a working cloud) and their requirements in terms of data storage. TL; DR: I still think in some cases we need to land containers on the same slave after upgrade, please provide your feedback. Thanks! -marek - [G.O] As I remember the spec for Mesos assumes that self-configuring service will be used. There is another spec for oslo.config to support remote configuration storages like ZooKeeper, ETCd, Consul. This approach should simplify an upgrade process as most of the configuration part will be done automatically by the service container itself. I think, we need to discuss the ways how OpenStack service can be upgraded and provide a baseline standards(requirements) for OpenStack services so that OpenStack service code will support one or another way for upgrades. Marathon framewok should support at least two ways of upgrades:(https://mesosphere.github.io/marathon/docs/deployment-design-doc.html) <https://mesosphere.github.io/marathon/docs/deployment-design-doc.html%29> 1) Rolling-Upgrade (Canary) 2) Green-Blue (A/B) upgrades As an operator I should be able to select the specific version of container which I want to roll-out to the existing cloud and I have to be able to do a rolling-back operation in case of upgrade failure. If we need to use volume based configuration storage then it should rely on Mesos Volume management (http://mesos.apache.org/documentation/latest/persistent-volume/) <http://mesos.apache.org/documentation/latest/persistent-volume/%29>which is not released yet as I know. Mesos/Marathon should be able to place upgared container correctly and we should not define any contstrains for that in the request. We still might use constraints but for providing more flexible/complex rolling-upgrade process like upgrading only specific number of instances at once. [M.Z.] I agree in general about Mesos maintaing upgrades but in some cases it's not about volumes but underlying (host-based) data (nova, cinder). Let's break it down into a list of OpenStack services (starting with a minimal list that is necessary to build a working cloud) and their requirements in terms of data storage: 1. nova: we need to ensure upgraded nova container is started on the same slave so it can reconnect to hypervisor and see the VMs. Not a candidate for Mesos Volumes (MVs). 2. cinder: must be on the same host to see block devices it uses for storage. Not a candidate for MVs. 3. mariadb: must be on the same host to see directory it uses for data. In the future when we'll use gallera perhaps it may be a group of hosts, not just 1 host. Sounds like a candidate for MVs. 4. keystone: does not care about the host 5. neutron: does not care about the host Additional notes: - currently for neutron we mount "/lib/modules" and do modprobe from the inside of the container to make neutron working - isnt' this wrong by design? Slave host should be prepared beforehand and load necessary modules. - since for now config files are stored on slaves in a well-known path (this is provisioned by ansible) we can assume each slave is identical in this regard so we can easily move services that do not use data storage between slaves * - also I don't see how we can avoid sticking to the same slave after upgrade (whatever rolling or green/blue) without MVs for mariadb and for nova & cinder/local disks at all. * - I can see cinder container not being host-dependent as long as it's not using local disks for storage (but ceph for example) On 16.12.2015 18:16, Marek Zawadzki wrote: Hello all, I described use case and simple tool that I'd like to implement as a first step in this topic - would you please review it and provide me with feedback before I start coding? Is the use-case realistic? Is this tool going to be useful given the features I described? Any other comments? https://etherpad.openstack.org/p/kolla_upgrading_containers Thank you, -marek zawadzki -- Marek Zawadzki Mirantis Kolla Team __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [kolla][nova] Orchiestrated upgrades in kolla
Michal, hi, what's your assumption about instances? Do we want to shut them down, migrate nova, and restart them or we'd like to preserve all running VMs? -marek On 04.12.2015 21:50, Michał Jastrzębski wrote: Hey guys, Orchiestrated upgrades is one of our highest priorities for M in kolla, so following up after discussion on summit I'd like to suggest an approach: Instead of creating playbook called "upgrade my openstack" we will create "upgrade my nova" instead and approach to each service case by case (since all of our running services are in dockers, this is possible). We will also make use of image tags as version carriers, so ansible will deploy new container only if version tag differs from what we ask it to deploy. This will help with idempotency of upgrade process. So let's start with nova. Upgrade-my-nova playbook will do something like this: 0. We create snapshot of our mariadb-data container. This will affect every service, but it's always good to have backup and rollback of db will be manual action 1. Nova bootstrap will be called and it will perform db-migration. Since current approach to nova code is add-only we shouldn't need to stop services and old services should keep working on newer database. Also for minor version upgrades there will be no action here unless there is migration. 2. We upgrade all conductor at the same time. This should take mere seconds since we'll have prebuilt containers 3. We will upgrade rest of controller services with using "series: 1" in ansible to ensure rolling upgrades. 4. We will upgrade all of nova-compute services on it's own pace. This workflow should be pretty robust (I think it is) and it should also provide idempotency. Thoughts? Regards, Michal __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Marek Zawadzki Mirantis Kolla Team __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [kolla] Upgrading containers in Mesos/Marathon environment - request for feedback
Hello all, I described use case and simple tool that I'd like to implement as a first step in this topic - would you please review it and provide me with feedback before I start coding? Is the use-case realistic? Is this tool going to be useful given the features I described? Any other comments? https://etherpad.openstack.org/p/kolla_upgrading_containers Thank you, -marek zawadzki -- Marek Zawadzki Mirantis Kolla Team __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
