Re: [openstack-dev] [TripleO][Edge] Reduce base layer of containers for security and size of images (maintenance) sakes

[Jiří Stránský]

On 29. 11. 18 20:20, Fox, Kevin M wrote:

If the base layers are shared, you won't pay extra for the separate puppet 
container


Yes, and that's the state we're in right now.


unless you have another container also installing ruby in an upper layer.


Not just Ruby but also Puppet and Systemd. I think that's what the 
proposal we're discussing here suggests -- removing this content from 
the base layer (so that we can get service runtime images without this 
content present) and putting this content *on top* of individual service 
images. Unless i'm missing some trick to start sharing *top* layers 
rather than *base* layers, i think that effectively disables the space 
sharing for the Ruby+Puppet+Systemd content.



With OpenStack, thats unlikely.

the apparent size of a container is not equal to its actual size.


Yes. :)

Thanks

Jirka



Thanks,
Kevin

From: Jiří Stránský [ji...@redhat.com]
Sent: Thursday, November 29, 2018 9:42 AM
To: openstack-dev@lists.openstack.org
Subject: Re: [openstack-dev] [TripleO][Edge] Reduce base layer of containers 
for security and size of images (maintenance) sakes

On 28. 11. 18 18:29, Bogdan Dobrelya wrote:

On 11/28/18 6:02 PM, Jiří Stránský wrote:





Reiterating again on previous points:

-I'd be fine removing systemd. But lets do it properly and not via 'rpm
-ev --nodeps'.
-Puppet and Ruby *are* required for configuration. We can certainly put
them in a separate container outside of the runtime service containers
but doing so would actually cost you much more space/bandwidth for each
service container. As both of these have to get downloaded to each node
anyway in order to generate config files with our current mechanisms
I'm not sure this buys you anything.


+1. I was actually under the impression that we concluded yesterday on
IRC that this is the only thing that makes sense to seriously consider.
But even then it's not a win-win -- we'd gain some security by leaner
production images, but pay for it with space+bandwidth by duplicating
image content (IOW we can help achieve one of the goals we had in mind
by worsening the situation w/r/t the other goal we had in mind.)

Personally i'm not sold yet but it's something that i'd consider if we
got measurements of how much more space/bandwidth usage this would
consume, and if we got some further details/examples about how serious
are the security concerns if we leave config mgmt tools in runtime images.

IIRC the other options (that were brought forward so far) were already
dismissed in yesterday's IRC discussion and on the reviews. Bin/lib bind
mounting being too hacky and fragile, and nsenter not really solving the
problem (because it allows us to switch to having different bins/libs
available, but it does not allow merging the availability of bins/libs
from two containers into a single context).



We are going in circles here I think


+1. I think too much of the discussion focuses on "why it's bad to have
config tools in runtime images", but IMO we all sorta agree that it
would be better not to have them there, if it came at no cost.

I think to move forward, it would be interesting to know: if we do this
(i'll borrow Dan's drawing):

|base container| --> |service container| --> |service container w/
Puppet installed|

How much more space and bandwidth would this consume per node (e.g.
separately per controller, per compute). This could help with decision
making.


As I've already evaluated in the related bug, that is:

puppet-* modules and manifests ~ 16MB
puppet with dependencies ~61MB
dependencies of the seemingly largest a dependency, systemd ~190MB

that would be an extra layer size for each of the container images to be
downloaded/fetched into registries.


Thanks, i tried to do the math of the reduction vs. inflation in sizes
as follows. I think the crucial point here is the layering. If we do
this image layering:

|base| --> |+ service| --> |+ Puppet|

we'd drop ~267 MB from base image, but we'd be installing that to the
topmost level, per-component, right?

In my basic deployment, undercloud seems to have 17 "components" (49
containers), overcloud controller 15 components (48 containers), and
overcloud compute 4 components (7 containers). Accounting for overlaps,
the total number of "components" used seems to be 19. (By "components"
here i mean whatever uses a different ConfigImage than other services. I
just eyeballed it but i think i'm not too far off the correct number.)

So we'd subtract 267 MB from base image and add that to 19 leaf images
used in this deployment. That means difference of +4.8 GB to the current
image sizes. My /var/lib/registry dir on undercloud with all the images
currently has 5.1 GB. We'd almost double that to 9.9 GB.

Going from 5.1 to 9.9 GB seems like a lot of extra traffic for the CDNs
(both external and e.g. internal within OpenStack Infra CI clouds).

And for interna

Re: [openstack-dev] [TripleO][Edge] Reduce base layer of containers for security and size of images (maintenance) sakes

[Jiří Stránský]

On 28. 11. 18 18:29, Bogdan Dobrelya wrote:

On 11/28/18 6:02 PM, Jiří Stránský wrote:





Reiterating again on previous points:

-I'd be fine removing systemd. But lets do it properly and not via 'rpm
-ev --nodeps'.
-Puppet and Ruby *are* required for configuration. We can certainly put
them in a separate container outside of the runtime service containers
but doing so would actually cost you much more space/bandwidth for each
service container. As both of these have to get downloaded to each node
anyway in order to generate config files with our current mechanisms
I'm not sure this buys you anything.


+1. I was actually under the impression that we concluded yesterday on
IRC that this is the only thing that makes sense to seriously consider.
But even then it's not a win-win -- we'd gain some security by leaner
production images, but pay for it with space+bandwidth by duplicating
image content (IOW we can help achieve one of the goals we had in mind
by worsening the situation w/r/t the other goal we had in mind.)

Personally i'm not sold yet but it's something that i'd consider if we
got measurements of how much more space/bandwidth usage this would
consume, and if we got some further details/examples about how serious
are the security concerns if we leave config mgmt tools in runtime images.

IIRC the other options (that were brought forward so far) were already
dismissed in yesterday's IRC discussion and on the reviews. Bin/lib bind
mounting being too hacky and fragile, and nsenter not really solving the
problem (because it allows us to switch to having different bins/libs
available, but it does not allow merging the availability of bins/libs
from two containers into a single context).



We are going in circles here I think


+1. I think too much of the discussion focuses on "why it's bad to have
config tools in runtime images", but IMO we all sorta agree that it
would be better not to have them there, if it came at no cost.

I think to move forward, it would be interesting to know: if we do this
(i'll borrow Dan's drawing):

|base container| --> |service container| --> |service container w/
Puppet installed|

How much more space and bandwidth would this consume per node (e.g.
separately per controller, per compute). This could help with decision
making.


As I've already evaluated in the related bug, that is:

puppet-* modules and manifests ~ 16MB
puppet with dependencies ~61MB
dependencies of the seemingly largest a dependency, systemd ~190MB

that would be an extra layer size for each of the container images to be
downloaded/fetched into registries.


Thanks, i tried to do the math of the reduction vs. inflation in sizes 
as follows. I think the crucial point here is the layering. If we do 
this image layering:


|base| --> |+ service| --> |+ Puppet|

we'd drop ~267 MB from base image, but we'd be installing that to the 
topmost level, per-component, right?


In my basic deployment, undercloud seems to have 17 "components" (49 
containers), overcloud controller 15 components (48 containers), and 
overcloud compute 4 components (7 containers). Accounting for overlaps, 
the total number of "components" used seems to be 19. (By "components" 
here i mean whatever uses a different ConfigImage than other services. I 
just eyeballed it but i think i'm not too far off the correct number.)


So we'd subtract 267 MB from base image and add that to 19 leaf images 
used in this deployment. That means difference of +4.8 GB to the current 
image sizes. My /var/lib/registry dir on undercloud with all the images 
currently has 5.1 GB. We'd almost double that to 9.9 GB.


Going from 5.1 to 9.9 GB seems like a lot of extra traffic for the CDNs 
(both external and e.g. internal within OpenStack Infra CI clouds).


And for internal traffic between local registry and overcloud nodes, it 
gives +3.7 GB per controller and +800 MB per compute. That may not be so 
critical but still feels like a considerable downside.


Another gut feeling is that this way of image layering would take longer 
time to build and to run the modify-image Ansible role which we use in 
CI, so that could endanger how our CI jobs fit into the time limit. We 
could also probably measure this but i'm not sure if it's worth spending 
the time.


All in all i'd argue we should be looking at different options still.



Given that we should decouple systemd from all/some of the dependencies
(an example topic for RDO [0]), that could save a 190MB. But it seems we
cannot break the love of puppet and systemd as it heavily relies on the
latter and changing packaging like that would higly likely affect
baremetal deployments with puppet and systemd co-operating.


Ack :/



Long story short, we cannot shoot both rabbits with a single shot, not
with puppet :) May be we could with ansible replacing puppet fully...
So splitting config and runtime images is the only choice yet to address
the raised 

Re: [openstack-dev] [TripleO][Edge] Reduce base layer of containers for security and size of images (maintenance) sakes

[Jiří Stránský]

Reiterating again on previous points:

-I'd be fine removing systemd. But lets do it properly and not via 'rpm
-ev --nodeps'.
-Puppet and Ruby *are* required for configuration. We can certainly put
them in a separate container outside of the runtime service containers
but doing so would actually cost you much more space/bandwidth for each
service container. As both of these have to get downloaded to each node
anyway in order to generate config files with our current mechanisms
I'm not sure this buys you anything.


+1. I was actually under the impression that we concluded yesterday on 
IRC that this is the only thing that makes sense to seriously consider. 
But even then it's not a win-win -- we'd gain some security by leaner 
production images, but pay for it with space+bandwidth by duplicating 
image content (IOW we can help achieve one of the goals we had in mind 
by worsening the situation w/r/t the other goal we had in mind.)


Personally i'm not sold yet but it's something that i'd consider if we 
got measurements of how much more space/bandwidth usage this would 
consume, and if we got some further details/examples about how serious 
are the security concerns if we leave config mgmt tools in runtime images.


IIRC the other options (that were brought forward so far) were already 
dismissed in yesterday's IRC discussion and on the reviews. Bin/lib bind 
mounting being too hacky and fragile, and nsenter not really solving the 
problem (because it allows us to switch to having different bins/libs 
available, but it does not allow merging the availability of bins/libs 
from two containers into a single context).




We are going in circles here I think


+1. I think too much of the discussion focuses on "why it's bad to have 
config tools in runtime images", but IMO we all sorta agree that it 
would be better not to have them there, if it came at no cost.


I think to move forward, it would be interesting to know: if we do this 
(i'll borrow Dan's drawing):


|base container| --> |service container| --> |service container w/
Puppet installed|

How much more space and bandwidth would this consume per node (e.g. 
separately per controller, per compute). This could help with decision 
making.




Dan



Thanks

Jirka

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo] Proposing Enrique Llorente Pastora as a core reviewer for TripleO

[Jiří Stránský]

On 15. 11. 18 16:54, Wesley Hayutin wrote:

On Thu, Nov 15, 2018 at 8:52 AM Sagi Shnaidman  wrote:


Hi,
I'd like to propose Quique (@quiquell) as a core reviewer for TripleO.
Quique is actively involved in improvements and development of TripleO and
TripleO CI. He also helps in other projects including but not limited to
Infrastructure.
He shows a very good understanding how TripleO and CI works and I'd like
suggest him as core reviewer of TripleO in CI related code.

Please vote!
My +1 is here :)



+1 for tripleo-ci core, I don't think we're proposing tripleo core atm.
Thanks for proposing and sending this Sagi!


+1!






Thanks
--
Best regards
Sagi Shnaidman
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo] [quickstart] [networking-ovn] No more overcloud_prep-containers.sh script

[Jiří Stránský]

Yes I know, but based on the deployment details we have for networking-ovn

it should be enough, we will have to update those documents with the new
changes anyway, because surprisingly this change has came for "Rocky" last
minute. Why did we have such last minute change? :-/

I understand the value of simplifying workflows to cloud operators, but
when we make workflow changes last minute we make others life harder (now I
need to rework something I want to be available in rocky, the migration
scripts/document).


Yea it's hard to strike a good balance there. It threw off updates and 
upgrades too. The upload is internally done via external_deploy_tasks 
which must not be run during `upgrade prepare` and `upgrade run` (or 
`update prepare` and `update run`). So likely Rocky updates/upgrades 
don't work as expected right now, and we'll need `external-upgrade run 
--tags image_prepare` inserted into the workflows. It's in my "queue of 
things to look into ASAP" ;D.


Jirka

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo] [quickstart] [networking-ovn] No more overcloud_prep-containers.sh script

[Jiří Stránský]

On 03/10/2018 10:14, Miguel Angel Ajo Pelayo wrote:

Hi folks

   I was trying to deploy neutron with networking-ovn via tripleo-quickstart
scripts on master, and this config file [1]. It doesn't work, overcloud
deploy cries with:

1) trying to deploy ovn I end up with a 2018-10-02 17:48:12 | "2018-10-02
17:47:51,864 DEBUG: 26691 -- Error: image
tripleomaster/centos-binary-ovn-controller:current-tripleo not found",

it seems like the overcloud_prep-containers.sh is not there anymore (I
guess overcloud deploy handles it automatically now? but it fails to
generate the ovn containers for some reason)

Also, if you look at [2] which are our ansible migration scripts to migrate
ml2/ovs to ml2/networking-ovn, you will see that we make use of
overcloud_prep-containers.sh , I guess that we will need to make sure [1]
works and we will get [2] for free.


Hi Miguel,

i'm not subject matter expert but here's some relevant info:

* overcloud_prep-containers.sh is not a production thing, it's 
automation from TripleO Quickstart, which is not part of production 
deployments. We shouldn't depend on it in docs/automation for OVN migration.


* For production envs, the image preparation steps used to be documented 
and performed manually. This now is now changing in Rocky+, as Steve 
Baker integrated the image prep into the deployment itself. There are 
docs about the current method [3].


* I hit similar issues with incorrect Neutron images being uploaded to 
undercloud registry, you can try deploying with this patch [4] which 
aims to fix that problem (also the depends-on patch is necessary).


Jirka


[1] https://github.com/openstack/networking-ovn/blob/master/tripleo/ovn.yml
[2] https://docs.openstack.org/networking-ovn/latest/install/migration.html


[3] 
http://tripleo.org/install/advanced_deployment/container_image_prepare.html

[4] https://review.openstack.org/#/c/604953/

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO] Removing global bootstrap_nodeid?

[Jiří Stránský]

Hi Steve,

On 25/09/2018 10:51, Steven Hardy wrote:

Hi all,

After some discussions with bandini at the PTG, I've been taking a
look at this bug and how to solve it:

https://bugs.launchpad.net/tripleo/+bug/1792613
(Also more information in downstream bz1626140)

The problem is that we always run container bootstrap tasks (as well
as a bunch of update/upgrade tasks) on the bootstrap_nodeid, which by
default is always the overcloud-controller-0 node (regardless of which
services are running there).

This breaks a pattern we established a while ago for Composable HA,
where we' work out the bootstrap node by
$service_short_bootstrap_hostname, which means we always run on the
first node that has the service enabled (even if it spans more than
one Role).

This presents two problems:

1. service_config_settings only puts the per-service hieradata on
nodes where a service is enabled, hence data needed for the
bootstrapping (e.g keystone users etc) can be missing if, say,
keystone is running on some role that's not Controller (this, I think
is the root-cause of the bug/bz linked above)

2. Even when we by luck have the data needed to complete the bootstrap
tasks, we'll end up pulling service containers to nodes where the
service isn't running, potentially wasting both time and space.

I've been looking at solutions, and it seems like we either have to
generate per-service bootstrap_nodeid's (I have a patch to do this
https://review.openstack.org/605010), or perhaps we could just remove
all the bootstrap node id's, and switch to using hostnames instead?
Seems like that could be simpler, but wanted to check if there's
anything I'm missing?


I think we should recheck he initial assumptions, because based on my 
testing:


* the bootstrap_nodeid is in fact a hostname already, despite its 
deceptive name,


* it's not global, it is per-role.

From my env:

[root@overcloud-controller-2 ~]# hiera -c /etc/puppet/hiera.yaml 
bootstrap_nodeid

overcloud-controller-0

[root@overcloud-novacompute-1 ~]# hiera -c /etc/puppet/hiera.yaml 
bootstrap_nodeid

overcloud-novacompute-0

This makes me think the problems (1) and (2) as stated above shouldn't 
be happening. The containers or tasks present in service definition 
should be executed on all nodes where a service is present, and if they 
additionally filter for bootstrap_nodeid, it would only pick one node 
per role. So, the service *should* be deployed on the selected bootstrap 
node, which means the service-specific hiera should be present there and 
needless container downloading should not be happening, AFAICT.


However, thinking about it this way, we probably have a different 
problem still:


(3) The actions which use bootstrap_nodeid check are not guaranteed to 
execute once per service. In case the service is present on multiple 
roles, the bootstrap_nodeid check succeeds once per such role.


Using per-service bootstrap host instead of per-role bootstrap host 
sounds like going the right way then.


However, none of the above provides a solid explanation of what's really 
happening in the LP/BZ mentioned above. Hopefully this info is at least 
a piece of the puzzle.


Jirka

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [openstack-ansible][kolla-ansible][tripleo] ansible roles: where they live and what do they do

[Jiří Stránský]

On 11.9.2018 18:53, Alex Schultz wrote:

Thanks everyone for coming and chatting.  From the meeting we've had a
few items where we can collaborate together.

Here are some specific bullet points:

- TripleO folks should feel free to propose some minor structural
changes if they make the integration easier.  TripleO is currently
investigating what it would look like to pull the keystone ansible
parts out of tripleo-heat-templates and put it into
ansible-role-tripleo-keystone.  It would be beneficial to use this
role as an example for how the os_keystone role can be consumed.


+1, please let's also focus on information flow towards Upgrades squad 
(and collab if needed) as ansible-role-tripleo-keystone is being 
implemented. It sounds like something that might potentially require 
rethinking how updates/upgrades work too. Maybe we could have a spec 
where the solution is described and we can assess/discuss the upgrades 
impact.


Thanks

Jirka


- The openstack-ansible-tests has some good examples of ansible-lint
rules that can be used to improve quality
- Tags could be used to limit the scope of OpenStack Ansible roles,
but it sounds like including tasks would be a better pattern.
- Need to establish a pattern for disabling packaging/service
configurations globally in OpenStack Ansible roles.
- Shared roles are open for reuse/replacement if something better is
available (upstream/elsewhere).

If anyone has any others, feel free to comment.

Thanks,
-Alex

On Mon, Sep 10, 2018 at 10:58 AM, Alex Schultz  wrote:

I just realized I booked the room and put it in the etherpad but
forgot to email out the time.

Time: Tuesday 09:00-10:45
Room: Big Thompson

https://etherpad.openstack.org/p/ansible-collaboration-denver-ptg

Thanks,
-Alex

On Tue, Sep 4, 2018 at 1:03 PM, Alex Schultz  wrote:

On Thu, Aug 9, 2018 at 2:43 PM, Mohammed Naser  wrote:

Hi Alex,

I am very much in favour of what you're bringing up.  We do have
multiple projects that leverage Ansible in different ways and we all
end up doing the same thing at the end.  The duplication of work is
not really beneficial for us as it takes away from our use-cases.

I believe that there is a certain number of steps that we all share
regardless of how we deploy, some of the things that come up to me
right away are:

- Configuring infrastructure services (i.e.: create vhosts for service
in rabbitmq, create databases for services, configure users for
rabbitmq, db, etc)
- Configuring inter-OpenStack services (i.e. keystone_authtoken
section, creating endpoints, etc and users for services)
- Configuring actual OpenStack services (i.e.
/etc//.conf file with the ability of extending
options)
- Running CI/integration on a cloud (i.e. common role that literally
gets an admin user, password and auth endpoint and creates all
resources and does CI)

This would deduplicate a lot of work, and especially the last one, it
might be beneficial for more than Ansible-based projects, I can
imagine Puppet OpenStack leveraging this as well inside Zuul CI
(optionally)... However, I think that this something which we should
discus further for the PTG.  I think that there will be a tiny bit
upfront work as we all standarize but then it's a win for all involved
communities.

I would like to propose that deployment tools maybe sit down together
at the PTG, all share how we use Ansible to accomplish these tasks and
then perhaps we can work all together on abstracting some of these
concepts together for us to all leverage.



I'm currently trying to get a spot on Tuesday morning to further
discuss some of this items.  In the mean time I've started an
etherpad[0] to start collecting ideas for things to discuss.  At the
moment I've got the tempest role collaboration and some basic ideas
for best practice items that we can discuss.  Feel free to add your
own and I'll update the etherpad with a time slot when I get one
nailed down.

Thanks,
-Alex

[0] https://etherpad.openstack.org/p/ansible-collaboration-denver-ptg


I'll let others chime in as well.

Regards,
Mohammed

On Thu, Aug 9, 2018 at 4:31 PM, Alex Schultz  wrote:

Ahoy folks,

I think it's time we come up with some basic rules/patterns on where
code lands when it comes to OpenStack related Ansible roles and as we
convert/export things. There was a recent proposal to create an
ansible-role-tempest[0] that would take what we use in
tripleo-quickstart-extras[1] and separate it for re-usability by
others.   So it was asked if we could work with the openstack-ansible
team and leverage the existing openstack-ansible-os_tempest[2].  It
turns out we have a few more already existing roles laying around as
well[3][4].

What I would like to propose is that we as a community come together
to agree on specific patterns so that we can leverage the same roles
for some of the core configuration/deployment functionality while
still allowing for specific project specific customization.  What I've
noticed between all the project is that we have a few 

Re: [openstack-dev] [Tripleo] Automating role generation

[Jiří Stránský]

On 4.9.2018 08:13, Janki Chhatbar wrote:

Hi

I am looking to automate role file generation in TripleO. The idea is
basically for an operator to create a simple yaml file (operator.yaml, say)
listing services that are needed and then TripleO to generate
Controller.yaml enabling only those services that are mentioned.

For example:
operator.yaml
services:
 Glance
 OpenDaylight
 Neutron ovs agent


I'm not sure it's worth introducing a new file format as such, if the 
purpose is essentially to expand e.g. "Glance" into 
"OS::TripleO::Services::GlanceApi" and 
"OS::TripleO::Services::GlanceRegistry"? It would be another layer of 
indirection (additional mental work for the operator who wants to 
understand how things work), while the layer doesn't make too much 
difference in preparation of the role. At least that's my subjective view.




Then TripleO should
1. Fail because ODL and OVS agent are either-or services


+1 i think having something like this would be useful.


2. After operator.yaml is modified to remove Neutron ovs agent, it should
generate Controller.yaml with below content

ServicesDefault:
- OS::TripleO::Services::GlanceApi
- OS::TripleO::Services::GlanceRegistry
- OS::TripleO::Services::OpenDaylightApi
- OS::TripleO::Services::OpenDaylightOvs

Currently, operator has to manually edit the role file (specially when
deployed with ODL) and I have seen many instances of failing deployment due
to variations of OVS, OVN and ODL services enabled when they are actually
exclusive.


Having validations on the service list would be helpful IMO, e.g. "these 
services must not be in one deployment together", "these services must 
not be in one role together", "these services must be together", "we 
recommend this service to be in every role" (i'm thinking 
TripleOPackages, Ntp, ...) etc. But as mentioned above, i think it would 
be better if we worked directly with the "OS::TripleO::Services..." 
values rather than a new layer of proxy-values.


Additional random related thoughts:

* Operator should still be able to disobey what the validation suggests, 
if they decide so.


* Would be nice to have the info about particular services (e.g what 
can't be together) specified declaratively somewhere (TripleO's favorite 
thing in the world -- YAML?).


* We could start with just one type of validation, e.g. the mutual 
exclusivity rule for ODL vs. OVS, but would be nice to have the solution 
easily expandable for new rule types.


Thanks for looking into this :)

Jirka



I am willing to spend some cycle on this. What I ask is some clearity on
its feasibility and any other ideas to make this idea into a feature.



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo] quickstart for humans

[Jiří Stránský]

On 30.8.2018 16:28, Honza Pokorny wrote:

Hello!

Over the last few months, it seems that tripleo-quickstart has evolved
into a CI tool.  It's primarily used by computers, and not humans.
tripleo-quickstart is a helpful set of ansible playbooks, and a
collection of feature sets.  However, it's become less useful for
setting up development environments by humans.  For example, devmode.sh
was recently deprecated without a user-friendly replacement. Moreover,
during some informal irc conversations in #oooq, some developers even
mentioned the plan to merge tripleo-quickstart and tripleo-ci.

I think it would be beneficial to create a set of defaults for
tripleo-quickstart that can be used to spin up new environments; a set
of defaults for humans.  This can either be a well-maintained script in
tripleo-quickstart itself, or a brand new project, e.g.
tripleo-quickstart-humans.  The number of settings, knobs, and flags
should be kept to a minimum.

This would accomplish two goals:

1.  It would bring uniformity to the team.  Each environment is
 installed the same way.  When something goes wrong, we can
 eliminate differences in setup when debugging.  This should save a
 lot of time.

2.  Quicker and more reliable environment setup.  If the set of defaults
 is used by many people, it should container fewer bugs because more
 people using something should translate into more bug reports, and
 more bug fixes.

These thoughts are coming from the context of tripleo-ui development.  I
need an environment in order to develop, but I don't necessarily always
care about how it's installed.  I want something that works for most
scenarios.

What do you think?  Does this make sense?  Does something like this
already exist?


I've been tinkering in this area for a long time, previously with 
inlunch [1], and now quicklunch [2] (which is a wrapper around 
quickstart), and i've been involved in various conversations about this 
over the past years, so i feel like i may have some insight to share on 
all this in general.


* A single config for everyone is not achievable, IMO. Someone wants HA, 
others want Ceph, Sahara, OpenDaylight, etc. There's no overlap here to 
be found i think, while respecting that the resulting deployment needs 
to be of reasonable size.


* "for humans" definition differs significantly based on who you ask. 
E.g. my intention with [2] was to readily expose *more* knobs and tweaks 
and be more transparent with the underlying workings of Ansible, because 
i felt like quickstart.sh hides too much from me. In my opinion [2] is 
sufficiently "for humans", yet it does pretty much the opposite of what 
you're looking for.


* It's hard to strike a good balance between for-CI and for-humans (and 
the various definitions of for-humans ;) ), but it's worth to keep doing 
that as high in the software stack as possible, because there is great 
value in running CI and all dev envs with the same (underlying) tool. 
Over the years i've observed that Quickstart is trying hard to 
consolidate various requirements, but it's simply difficult to please 
all stakeholders, as often the requirements are somewhat contradictory. 
(This point is not in conflict with anything discussed so far, but i 
just think it's worth mentioning, so that we don't display Quickstart in 
a way it doesn't deserve.)


These points are to illustrate my previous experience, that when we go 
above a certain layer of "this is a generic all-purpose configurable 
tool" (= Quickstart), it seems to yield better results to focus on 
building the next layer/wrapper for "humans like me" rather than "humans 
in general".


So with regards to the specific goal stemming from tripleo-ui dev 
requirements as you mentioned above, IMO it makes sense to team up with 
UI folks and others who have similar expectations about what a TripleO 
dev env means, and make some wrapper around Quickstart like you 
suggested. Since you want to reduce rather than extend the number of 
knobs, it could even be just a script perhaps.


My 2 cents, i hope it helps a bit.

Jirka

[1] https://github.com/jistr/inlunch
[2] https://gitlab.com/jistr/quicklunch



Thanks for listening!

Honza

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo][Edge][FEMDC] Edge clouds and controlplane updates

[Jiří Stránský]

On 14.8.2018 15:19, Bogdan Dobrelya wrote:

On 8/13/18 9:47 PM, Giulio Fidente wrote:

Hello,

I'd like to get some feedback regarding the remaining
work for the split controlplane spec implementation [1]

Specifically, while for some services like nova-compute it is not
necessary to update the controlplane nodes after an edge cloud is
deployed, for other services, like cinder (or glance, probably
others), it is necessary to do an update of the config files on the
controlplane when a new edge cloud is deployed.

In fact for services like cinder or glance, which are hosted in the
controlplane, we need to pull data from the edge clouds (for example
the newly deployed ceph cluster keyrings and fsid) to configure cinder
(or glance) with a new backend.

It looks like this demands for some architectural changes to solve the > 
following two:

- how do we trigger/drive updates of the controlplane nodes after the
edge cloud is deployed?


Note, there is also a strict(?) requirement of local management
capabilities for edge clouds temporary disconnected off the central
controlplane. That complicates the updates triggering even more. We'll
need at least a notification-and-triggering system to perform required
state synchronizations, including conflicts resolving. If that's the
case, the architecture changes for TripleO deployment framework are
inevitable AFAICT.


Indeed this would complicate things much, but IIUC the spec [1] that 
Giulio referenced doesn't talk about local management at all.


Within the context of what the spec covers, i.e. 1 stack for Controller 
role and other stack(s) for Compute or *Storage roles, i hope we could 
address updates/upgrades workflow similarly as the deployment workflow 
would be addressed -- working with the stacks one by one.


That would probably mean:

1. `update/upgrade prepare` on Controller stack

2. `update/upgrade prepare` on other stacks (perhaps reusing some 
outputs from Controller stack here)


3. `update/upgrade run` on Controller stack

4. `update/upgrade run` on other stacks

5. (`external-update/external-upgrade run` on other stacks where 
appropriate)


6. `update/upgrade converge` on Controller stack

7. `update/upgrade converge` on other stacks (again maybe reusing 
outputs from Controller stack)


I'm not *sure* such approach would work, but at the moment i don't see a 
reason why it wouldn't :)


Jirka





- how do we scale the controlplane parameters to accomodate for N
backends of the same type?

A very rough approach to the latter could be to use jinja to scale up
the CephClient service so that we can have multiple copies of it in the
controlplane.

Each instance of CephClient should provide the ceph config file and
keyring necessary for each cinder (or glance) backend.

Also note that Ceph is only a particular example but we'd need a similar
workflow for any backend type.

The etherpad for the PTG session [2] touches this, but it'd be good to
start this conversation before then.

1.
https://specs.openstack.org/openstack/tripleo-specs/specs/rocky/split-controlplane.html

2. https://etherpad.openstack.org/p/tripleo-ptg-queens-split-controlplane







__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO] podman: varlink interface for nice API calls

[Jiří Stránský]

On 16.8.2018 07:39, Cédric Jeanneret wrote:



On 08/16/2018 12:10 AM, Jason E. Rist wrote:

On 08/15/2018 03:32 AM, Cédric Jeanneret wrote:

Dear Community,

As you may know, a move toward Podman as replacement of Docker is starting.

One of the issues with podman is the lack of daemon, precisely the lack
of a socket allowing to send commands and get a "computer formatted
output" (like JSON or YAML or...).

In order to work that out, Podman has added support for varlink¹, using
the "socket activation" feature in Systemd.

On my side, I would like to push forward the integration of varlink in
TripleO deployed containers, especially since it will allow the following:
# proper interface with Paunch (via python link)

# a way to manage containers from within specific containers (think
"healthcheck", "monitoring") by mounting the socket as a shared volume

# a way to get container statistics (think "metrics")

# a way, if needed, to get an ansible module being able to talk to
podman (JSON is always better than plain text)

# a way to secure the accesses to Podman management (we have to define
how varlink talks to Podman, maybe providing dedicated socket with
dedicated rights so that we can have dedicated users for specific tasks)

That said, I have some questions:
° Does any of you have some experience with varlink and podman interface?
° What do you think about that integration wish?
° Does any of you have concern with this possible addition?

Thank you for your feedback and ideas.

Have a great day (or evening, or whatever suits the time you're reading
this ;))!

C.


¹ https://www.projectatomic.io/blog/2018/05/podman-varlink/




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



How might this effect upgrades?


What exactly? addition of varlink, or the whole podman thingy? The
question was more about "varlink" than "podman" in fact - I should maybe
have worded things otherwise... ?


Varlink shouldn't be a problem as it's just an additive interface. 
Switching container runtime might be a bit difficult though :)


When running any upgrade, we stop any containers that need updating, and 
replace them with new ones. In theory we could just as well start the 
new ones using a different runtime, all we need is to keep the same bind 
mounts etc. What would need to be investigated is whether support for 
this (stopping on one runtime, starting on another) needs to be 
implemented directly into tools like Paunch and Pacemaker, or if we can 
handle this one-time scenario just with additional code in 
upgrade_tasks. It might be a combination of both.


Problem might come with sidecar containers for Neutron, which generally 
don't like being restarted (it can induce data plane downtime). Advanced 
hackery might be needed on this front... :)


Either way i think we'd have to do some PoC of such migration before 
fully committing to it.


Jirka





-J





__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO] podman: varlink interface for nice API calls

[Jiří Stránský]

On 15.8.2018 11:32, Cédric Jeanneret wrote:

Dear Community,

As you may know, a move toward Podman as replacement of Docker is starting.

One of the issues with podman is the lack of daemon, precisely the lack
of a socket allowing to send commands and get a "computer formatted
output" (like JSON or YAML or...).

In order to work that out, Podman has added support for varlink¹, using
the "socket activation" feature in Systemd.

On my side, I would like to push forward the integration of varlink in
TripleO deployed containers, especially since it will allow the following:
# proper interface with Paunch (via python link)


"integration of varlink in TripleO deployed containers" sounds like we'd 
need to make some changes to the containers themselves, but is that the 
case? As i read the docs, it seems like a management API wrapper for 
Podman, so just an alternative interface to Podman CLI. I'd expect we'd 
use varlink from Paunch, but probably not from the containers 
themselves? (Perhaps that's what you meant, just making sure we're on 
the same page.)




# a way to manage containers from within specific containers (think
"healthcheck", "monitoring") by mounting the socket as a shared volume


I think healthchecks are currently quite Docker-specific, so we could 
have a Podman-specific alternative here. We should be careful about how 
much container runtime specificity we introduce and keep though, and 
we'll probably have to amend our tools (e.g. pre-upgrade validations 
[2]) to work with both, at least until we decide whether to really make 
a full transition to Podman or not.




# a way to get container statistics (think "metrics")

# a way, if needed, to get an ansible module being able to talk to
podman (JSON is always better than plain text)

# a way to secure the accesses to Podman management (we have to define
how varlink talks to Podman, maybe providing dedicated socket with
dedicated rights so that we can have dedicated users for specific tasks)

That said, I have some questions:
° Does any of you have some experience with varlink and podman interface?
° What do you think about that integration wish?
° Does any of you have concern with this possible addition?


I like it, but we should probably sync up with Podman community if they 
consider varlink a "supported" interface for controlling Podman, and 
it's not just an experiment which will vanish. To me it certainly looks 
like a much better programmable interface than composing CLI calls and 
parsing their output, but we should make sure Podman folks think so too :)


Thanks for looking into this

Jirka

[2] https://review.openstack.org/#/c/582502/



Thank you for your feedback and ideas.

Have a great day (or evening, or whatever suits the time you're reading
this ;))!

C.


¹ https://www.projectatomic.io/blog/2018/05/podman-varlink/




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO] podman: varlink interface for nice API calls

[Jiří Stránský]

On 16.8.2018 10:38, Steven Hardy wrote:

On Wed, Aug 15, 2018 at 10:48 PM, Jay Pipes  wrote:

On 08/15/2018 04:01 PM, Emilien Macchi wrote:


On Wed, Aug 15, 2018 at 5:31 PM Emilien Macchi mailto:emil...@redhat.com>> wrote:

 More seriously here: there is an ongoing effort to converge the
 tools around containerization within Red Hat, and we, TripleO are
 interested to continue the containerization of our services (which
 was initially done with Docker & Docker-Distribution).
 We're looking at how these containers could be managed by k8s one
 day but way before that we plan to swap out Docker and join CRI-O
 efforts, which seem to be using Podman + Buildah (among other things).

I guess my wording wasn't the best but Alex explained way better here:

http://eavesdrop.openstack.org/irclogs/%23openstack-tc/%23openstack-tc.2018-08-15.log.html#t2018-08-15T17:56:52

If I may have a chance to rephrase, I guess our current intention is to
continue our containerization and investigate how we can improve our tooling
to better orchestrate the containers.
We have a nice interface (openstack/paunch) that allows us to run multiple
container backends, and we're currently looking outside of Docker to see how
we could solve our current challenges with the new tools.
We're looking at CRI-O because it happens to be a project with a great
community, focusing on some problems that we, TripleO have been facing since
we containerized our services.

We're doing all of this in the open, so feel free to ask any question.



I appreciate your response, Emilien, thank you. Alex' responses to Jeremy on
the #openstack-tc channel were informative, thank you Alex.

For now, it *seems* to me that all of the chosen tooling is very Red Hat
centric. Which makes sense to me, considering Triple-O is a Red Hat product.


Just as a point of clarification - TripleO is an OpenStack project,
and yes there is a downstream product derived from it, but we could
e.g support multiple container backends in TripleO if there was
community interest in supporting that.

Also I think Alex already explained that fairly clearly in the IRC
link that this is initially about proving our existing abstractions
work to enable alternate container backends.


+1, and with my upgrade-centric hat on, we've had a fair share of 
trouble with Docker -- update of the daemon causing otherwise needless 
downtime of services and sometimes data plane too. Most recent example i 
can think of is here [1][2] -- satisfactory solution still doesn't 
exist. So my 2 cents: i am very interested in exploring alternative 
container runtimes, and daemon-less sounds to me like a promising direction.


Jirka

[1] https://bugs.launchpad.net/tripleo/+bug/1777146
[2] https://review.openstack.org/#/c/575758/1/puppet/services/docker.yaml



Thanks,

Steve

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo] Proposing Lukas Bezdicka core on TripleO

[Jiří Stránský]

+1!

On 1.8.2018 13:31, Giulio Fidente wrote:

Hi,

I would like to propose Lukas Bezdicka core on TripleO.

Lukas did a lot work in our tripleoclient, tripleo-common and
tripleo-heat-templates repos to make FFU possible.

FFU, which is meant to permit upgrades from Newton to Queens, requires
in depth understanding of many TripleO components (for example Heat,
Mistral and the TripleO client) but also of specific TripleO features
which were added during the course of the three releases (for example
config-download and upgrade tasks). I believe his FFU work to have been
very challenging.

Given his broad understanding, more recently Lukas started helping doing
reviews in other areas.

I am so sure he'll be a great addition to our group that I am not even
looking for comments, just votes :D




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [tripleo] Rocky Ceph update/upgrade regression risk (semi-FFE)

[Jiří Stránský]

Hi folks,

i want to raise attention on remaining patches that are needed to 
prevent losing Ceph updates/upgrades in Rocky [1], basically making the 
Ceph upgrade mechanism compatible with config-download. I'd call this a 
semi-FFE, as a few of the patches have characteristics of feature work, 
but at the same time i don't believe we can afford having Ceph 
unupgradable in Rocky, so it has characteristics of a regression bug 
too. I reported a bug [2] and tagged the patches in case we end up 
having to do backports.


Please help with reviews and landing the patches if possible.


It would have been better to focus on this earlier in the cycle, but 
majority of Upgrades squad work is exactly this kind of semi-FFE -- 
nontrivial in terms of effort required, but at the same time it's not 
something we can realistically slip into the next release, because it 
would be a regression. This sort of work tends to steal some of our 
focus in N cycle and direct it towards N-1 release (or even older). 
However, i think we've been gradually catching up with the release cycle 
lately, and increased focus on keeping update/upgrade CI green helps us 
catch breakages before they land and saves some person-hours, so i'm 
hoping the future is bright(er) on this.



Thanks and have a good day,

Jirka

[1] https://review.openstack.org/#/q/topic:external-update-upgrade
[2] https://bugs.launchpad.net/tripleo/+bug/1783949

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo] Proposing Jose Luis Franco for TripleO core reviewer on Upgrade bits

[Jiří Stránský]

+1!

On 20.7.2018 10:07, Carlos Camacho Gonzalez wrote:

Hi!!!

I'll like to propose Jose Luis Franco [1][2] for core reviewer in all the
TripleO upgrades bits. He shows a constant and active involvement in
improving and fixing our updates/upgrades workflows, he helps also trying
to develop/improve/fix our upstream support for testing the
updates/upgrades.

Please vote -1/+1, and consider this my +1 vote :)

[1]: https://review.openstack.org/#/q/owner:jfrancoa%2540redhat.com
[2]: http://stackalytics.com/?release=all=commits_id=jfrancoa

Cheers,
Carlos.



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo] Updates/upgrades equivalent for external_deploy_tasks

[Jiří Stránský]

Thanks for the feedback, John and Emilien.

On 13.7.2018 01:35, Emilien Macchi wrote:

+1 for Option A as well, I feel like it's the one which would give us the
more of flexibility and also I'm not a big fan of the usage of Anchors for
this use case.
Some folks are currently working on extracting these tasks out of THT and I
can already see something like:

 external_deploy_tasks
   - include_role:
   name: my-service
   tasks_from: deploy

 external_update_tasks
   - include_role:
   name: my-service
   tasks_from: update

Or we could re-use the same playbooks, but use tags maybe.
Anyway, I like your proposal and I vote for option A.


I like the tasks_from approach in the snippet. Regarding tags, i'm 
currently thinking of using them to optionally update/upgrade individual 
services which make use of external_*_tasks. E.g. in an environment with 
both OpenShift and Ceph, i'm hoping we could run:


openstack overcloud external-update run --tags ceph
openstack overcloud external-update run --tags openshift

to update them separately if needed. That's the way i'm trying to 
prototype it right now anyway, open to feedback.


Jirka

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [tripleo] Updates/upgrades equivalent for external_deploy_tasks

[Jiří Stránský]

Hi,

with the move to config-download deployments, we'll be moving from 
executing external installers (like ceph-ansible) via Heat resources 
encapsulating Mistral workflows towards executing them via Ansible 
directly (nested Ansible process via external_deploy_tasks).


Updates and upgrades still need to be addressed here. I think we should 
introduce external_update_tasks and external_upgrade_tasks for this 
purpose, but i see two options how to construct the workflow with them.


During update (mentioning just updates, but upgrades would be done 
analogously) we could either:


A) Run external_update_tasks, then external_deploy_tasks.

This works with the assumption that updates are done very similarly to 
deployment. The external_update_tasks could do some prep work and/or 
export Ansible variables which then could affect what 
external_deploy_tasks do (e.g. in case of ceph-ansible we'd probably 
override the playbook path). This way we could also disable specific 
parts of external_deploy_tasks on update, in case reuse is undesirable 
in some places.


B) Run only external_update_tasks.

This would mean code for updates/upgrades of externally deployed 
services would be completely separated from how their deployment is 
done. If we wanted to reuse some of the deployment tasks, we'd have to 
use the YAML anchor referencing mechanisms. (, *anchor)


I think the options are comparable in terms of what is possible to 
implement with them, the main difference is what use cases we want to 
optimize for.


Looking at what we currently have in external_deploy_tasks (e.g. 
[1][2]), i think we'd have to do a lot of explicit reuse if we went with 
B (inventory and variables generation, ...). So i'm leaning towards 
option A (WIP patch at [3]) which should give us this reuse more 
naturally. This approach would also be more in line with how we already 
do normal updates and upgrades (also reusing deployment tasks). Please 
let me know in case you have any concerns about such approach (looking 
especially at Ceph and OpenShift integrators :) ).


Thanks

Jirka

[1] 
https://github.com/openstack/tripleo-heat-templates/blob/8d7525fdf79f915e3f880ea0f3fd299234ecc635/docker/services/ceph-ansible/ceph-base.yaml#L340-L467
[2] 
https://github.com/openstack/tripleo-heat-templates/blob/8d7525fdf79f915e3f880ea0f3fd299234ecc635/extraconfig/services/openshift-master.yaml#L70-L231

[3] https://review.openstack.org/#/c/579170/

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] DeployArtifacts considered...complicated?

[Jiří Stránský]

On 19.6.2018 16:29, Lars Kellogg-Stedman wrote:

On Tue, Jun 19, 2018 at 02:18:38PM +0100, Steven Hardy wrote:

Is this the same issue Carlos is trying to fix via
https://review.openstack.org/#/c/494517/ ?


That solves part of the problem, but it's not a complete solution.
In particular, it doesn't solve the problem that bit me: if you're
changing puppet providers (e.g., replacing
provider/keystone_config/ini_setting.rb with
provider/keystone_config/openstackconfig.rb), you still have the old
provider sitting around causing problems because unpacking a tarball
only *adds* files.


Yeah I think we've never seen this because normally the
/etc/puppet/modules tarball overwrites the symlink, effectively giving
you a new tree (the first time round at least).


But it doesn't, and that's the unexpected problem: if you replace the
/etc/puppet/modules/keystone symlink with a directory, then
/usr/share/openstack-puppet/modules/keystone is still there, and while
the manifests won't be used, the contents of the lib/ directory will
still be active.


Probably we could add something to the script to enable a forced
cleanup each update:

https://github.com/openstack/tripleo-heat-templates/blob/master/puppet/deploy-artifacts.sh#L9


We could:

(a) unpack the replacement puppet modules into a temporary location,
   then

(b) for each module; rm -rf the target directory and then copy it into
   place

But! This would require deploy_artifacts.sh to know that it was
unpacking puppet modules rather than a generic tarball.


This would have to be optional, so we could add something like a
DeployArtifactsCleanupDirs parameter perhaps?


If we went with the above, sure.


One more thought which just occurred to me - we could add support for
a git checkout/pull to the script?


Reiterating our conversion in #tripleo, I think rather than adding a
bunch of specific functionality to the DeployArtifacts feature, it
would make more sense to add the ability to include some sort of
user-defined pre/post tasks, either as shell scripts or as ansible
playbooks or something.

On the other hand, I like your suggestion of just ditching
DeployArtifacts for a new composable service that defines
host_prep_tasks (or re-implenting DeployArtifacts as a composable
service), so I'm going to look at that as a possible alternative to
what I'm currently doing.



For the puppet modules specifically, we might also add another 
directory+mount into the docker-puppet container, which would be blank 
by default (unlike the existing, already populated /etc/puppet and 
/usr/share/openstack-puppet/modules). And we'd put that directory at the 
very start of modulepath. Then i *think* puppet would use a particular 
module from that dir *only*, not merge the contents with the rest of 
modulepath, so stale files in /etc/... or /usr/share/... wouldn't matter 
(didn't test it though). That should get us around the "tgz only adds 
files" problem without any rm -rf.


The above is somewhat of an orthogonal suggestion to the composable 
service approach, they would work well alongside i think. (And +1 on 
"DeployArtifacts as composable service" as something worth investigating 
/ implementing.)


Jirka

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo] Proposing Alan Bishop tripleo core on storage bits

[Jiří Stránský]

+1

On 13.6.2018 17:50, Emilien Macchi wrote:

Alan Bishop has been highly involved in the Storage backends integration in
TripleO and Puppet modules, always here to update with new features, fix
(nasty and untestable third-party backends) bugs and manage all the
backports for stable releases:
https://review.openstack.org/#/q/owner:%22Alan+Bishop+%253Cabishop%2540redhat.com%253E%22

He's also well knowledgeable of how TripleO works and how containers are
integrated, I would like to propose him as core on TripleO projects for
patches related to storage things (Cinder, Glance, Swift, Manila, and
backends).

Please vote -1/+1,
Thanks!



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo] scenario000-multinode-oooq-container-upgrades

[Jiří Stránský]

On 12.6.2018 15:06, James Slagle wrote:

On Mon, Jun 11, 2018 at 3:34 PM, Wesley Hayutin  wrote:

Greetings,

I wanted to let everyone know that we have a keystone only deployment and
upgrade job in check non-voting.  I'm asking everyone in TripleO to be
mindful of this job and to help make sure it continues to pass as we move it
from non-voting check to check and eventually gating.


+1, nice work!


Upgrade jobs are particularly difficult to keep running successfully because
of the complex workflow itself, job run times and other factors.  Your help
to ensure we don't merge w/o a pass on this job will go a long way in
helping the tripleo upgrades team.

There is still work to be done here, however it's much easier to do it with
the check non-voting job in place.


The job doesn't appear to be passing at all on stable/queens. I see
this same failure on several patches:
http://logs.openstack.org/59/571459/1/check/tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades/8bbd827/logs/undercloud/home/zuul/overcloud_upgrade_run_Controller.log.txt.gz

Is this a known issue?


I think so, or to put it precisely, i only ever looked into making the 
job work for master (and beyond).


We could look into making it work on Queens too, but personally i think 
effort would be better spent elsewhere at this point. E.g. upd+upg jobs 
with more complete of services utilizing containerized undercloud (those 
would not validate OC workflow at all, but would give coverage for 
update_tasks/upgrade_tasks), user and dev docs around all lifecycle ops 
(upd, upg, ffwd), upgrade work in the area of TLS by default, upgrade 
handling for external_deploy_tasks (= "how do we upgrade Ceph in 
Rocky"), also perhaps trying to DRY repeated parts of upgrade templates, 
etc.


If someone wants to step up to iron out Queens issues with that job then 
we can do it, but my 2 cents would be just to disable the job on Queens 
and focus on the future.


Jirka

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [tripleo] Zuul repo insertion in update/upgrade CI

[Jiří Stránský]

Hi,

this is mainly for CI folks and whom-it-may-concern.

Recently we came across the topic of how to enable/disable zuul repos at 
various places in the CI jobs. For normal deploy jobs there's no need to 
customize, but for update/upgrade jobs there is. It's not entirely 
straightforward and there's quite a variety of enable/disable spots and 
combinations which can be useful.


Even though improvements in this area are not very likely to get 
implemented right away, i had some thoughts on the topic so i wanted to 
capture them. I put the ideas into an etherpad:


https://etherpad.openstack.org/p/tripleo-ci-zuul-repo-insertion

Feel free to put some more thoughts there or ping me on IRC with 
anything related.



Thanks

Jirka

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo] Proposing Marius Cornea core on upgrade bits

[Jiří Stránský]

+1!

On 19.4.2018 19:01, Emilien Macchi wrote:

Greetings,

As you probably know mcornea on IRC, Marius Cornea has been contributing on
TripleO for a while, specially on the upgrade bits.
Part of the quality team, he's always testing real customer scenarios and
brings a lot of good feedback in his reviews, and quite often takes care of
fixing complex bugs when it comes to advanced upgrades scenarios.
He's very involved in tripleo-upgrade repository where he's already core,
but I think it's time to let him +2 on other tripleo repos for the patches
related to upgrades (we trust people's judgement for reviews).

As usual, we'll vote!

Thanks everyone for your feedback and thanks Marius for your hard work and
involvement in the project.



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo][ci] use of tags in launchpad bugs

[Jiří Stránský]

On 5.4.2018 21:04, Alex Schultz wrote:

On Thu, Apr 5, 2018 at 12:55 PM, Wesley Hayutin  wrote:

FYI...

This is news to me so thanks to Emilien for pointing it out [1].
There are official tags for tripleo launchpad bugs.  Personally, I like what
I've seen recently with some extra tags as they could be helpful in finding
the history of particular issues.
So hypothetically would it be "wrong" to create an official tag for each
featureset config number upstream.  I ask because that is adding a lot of
tags but also serves as a good test case for what is good/bad use of tags.



We list official tags over in the specs repo[0].   That being said as
we investigate switching over to storyboard, we'll probably want to
revisit tags as they will have to be used more to replace some of the
functionality we had with launchpad (e.g. milestones).  You could
always add the tags without being an official tag. I'm not sure I
would really want all the featuresets as tags.  I'd rather see us
actually figure out what component is actually failing than relying on
a featureset (and the Rosetta stone for decoding featuresets to
functionality[1]).


We could also use both alongside. Component-based tags better relate to 
the actual root cause of the bug, while featureset-based tags are useful 
in relation to CI.


E.g. "I see fs037 failing, i wonder if anyone already reported a bug for 
it" -- if the reporter tagged the bug, it would be really easy to figure 
out the answer.


This might also again bring up the question of better job names to allow 
easier mapping to featuresets. IMO:


tripleo-ci-centos-7-containers-multinode  -- not great
tripleo-ci-centos-7-featureset010  -- not great
tripleo-ci-centos-7-containers-mn-fs010  -- *happy face*

Jirka




Thanks,
-Alex


[0] 
http://git.openstack.org/cgit/openstack/tripleo-specs/tree/specs/policy/bug-tagging.rst#n30
[1] 
https://git.openstack.org/cgit/openstack/tripleo-quickstart/tree/doc/source/feature-configuration.rst#n21

Thanks

[1] https://bugs.launchpad.net/tripleo/+manage-official-tags

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo] how to reproduce tripleo ci

[Jiří Stránský]

On 6.2.2018 19:58, Wesley Hayutin wrote:

Greetings,

The TripleO-CI team has added a recreate / reproduce script to all the
tripleo upstream ci jobs as an artifact [1-2] much like the devstack
reproduce.sh script.  If you find yourself in need of recreating a tripleo
ci job please take a look at the instructions.


That looks great, kudos CI team!



At this time the reproduction of ci is only supported by using an openstack
cloud to provision test nodes, libvirt and other approaches are not yet
supported but are on the roadmap.


If someone needs to deploy on libvirt something resembling CI, it seems 
generally possible to use the featuresets as general_config for OOOQ on 
libvirt too (with small tweaks). I've used featuresets to deploy 
overcloud (on a pre-existing reusable undercloud) when working on 
Kubernetes/OpenShift [3-4].


The main thing to watch out for is that the featuresets may be setting 
NIC configs unsuitable for libvirt env. I think those are set either via 
the `network_isolation_args` parameter directly in the featureset, or 
via referencing a t-h-t scenario which includes 
`OS::TripleO::*::Net::SoftwareConfig` overrides in its resource 
registry. So one has to make sure to override those overrides. :)


Jirka



Thank you!

[1]
https://docs.openstack.org/tripleo-docs/latest/contributor/reproduce-ci.html
[2] http://tripleo.org/contributor/reproduce-ci.html

[3] https://www.jistr.com/blog/2017-11-21-kubernetes-in-tripleo/
[4] https://www.jistr.com/blog/2018-01-04-openshift-origin-in-tripleo/




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo] Proposing Wesley Hayutin core on TripleO CI

[Jiří Stránský]

+1

On 6.12.2017 16:45, Emilien Macchi wrote:

Team,

Wes has been consistently and heavily involved in TripleO CI work.
He has a very well understanding on how tripleo-quickstart and
tripleo-quickstart-extras work, his number and quality of reviews are
excellent so far. His experience with testing TripleO is more than
valuable.
Also, he's always here to help on TripleO CI issues or just
improvements (he's the guy filling bugs on a Saturday evening).
I think he would be a good addition to the TripleO CI core team
(tripleo-ci, t-q and t-q-e repos for now).
Anyway, thanks a lot Wes for your hard work on CI, I think it's time
to move on and get you +2 ;-)

As usual, it's open for voting, feel free to bring any feedback.
Thanks everyone,




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo] Updates on the TripleO on Kubernetes work

[Jiří Stránský]

On 21.11.2017 12:01, Jiří Stránský wrote:

Kubernetes on the overcloud
===

The work on this front started with 2[0][1] patches that some of you might have
seen and then evolved into using the config download mechanism to execute these
tasks as part of the undercloud tasks[2][3] (Thanks a bunch, Jiri, for your work
here). Note that [0] needs to be refactored to use the same mechanism used in
[2].


For those interested in trying the work we've done on deploying vanilla
Kubernetes, i put together a post showing how to deploy it with OOOQ,
and also briefly explaining the new external_deploy_tasks in service
templates:

https://www.jistr.com/blog/2017-11-21-kubernetes-in-tripleo/


And we've had a first Kubespray deployment success in CI, the job is 
ready to move from experimental to non-voting check [1]. The job doesn't 
yet deploy any pods on that Kubernetes cluster, but it's a step ;)


[1] https://review.openstack.org/#/c/524547/





There are quite a few things to improve here:

- How to configure/manage the loadbalancer/vips on the overcloud Kubespray is
- currently being cloned and we need to build a package for it More CI is likely
- needed for this work

[0] https://review.openstack.org/494470
[1] https://review.openstack.org/471759
[2] https://review.openstack.org/#/c/511272/
[3] https://review.openstack.org/#/c/514730/


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO] Proposing Ronelle Landy for Tripleo-Quickstart/Extras/CI core

[Jiří Stránský]

+1

On 29.11.2017 20:34, John Trowbridge wrote:

I would like to propose Ronelle be given +2 for the above repos. She has
been a solid contributor to tripleo-quickstart and extras almost since the
beginning. She has solid review numbers, but more importantly has always
done quality reviews. She also has been working in the very intense rover
role on the CI squad in the past CI sprint, and has done very well in that
role.



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo] Updates on the TripleO on Kubernetes work

[Jiří Stránský]

Kubernetes on the overcloud
===

The work on this front started with 2[0][1] patches that some of you might have
seen and then evolved into using the config download mechanism to execute these
tasks as part of the undercloud tasks[2][3] (Thanks a bunch, Jiri, for your work
here). Note that [0] needs to be refactored to use the same mechanism used in
[2].


For those interested in trying the work we've done on deploying vanilla 
Kubernetes, i put together a post showing how to deploy it with OOOQ, 
and also briefly explaining the new external_deploy_tasks in service 
templates:


https://www.jistr.com/blog/2017-11-21-kubernetes-in-tripleo/



There are quite a few things to improve here:

- How to configure/manage the loadbalancer/vips on the overcloud Kubespray is
- currently being cloned and we need to build a package for it More CI is likely
- needed for this work

[0] https://review.openstack.org/494470
[1] https://review.openstack.org/471759
[2] https://review.openstack.org/#/c/511272/
[3] https://review.openstack.org/#/c/514730/


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo] Proposing John Fulton core on TripleO

[Jiří Stránský]

+1!

On 8.11.2017 23:24, Giulio Fidente wrote:

Hi,

I would like to propose John Fulton core on TripleO.

I think John did an awesome work during the Pike cycle around the
integration of ceph-ansible as a replacement for puppet-ceph, for the
deployment of Ceph in containers.

I think John has good understanding of many different parts of TripleO
given that the ceph-ansible integration has been a complicated effort
involving changes in heat/tht/mistral workflows/ci and last but not
least, docs and he is more recently getting busier with reviews outside
his main comfort zone.

I am sure John would be a great addition to the team and I welcome him
first to tune into radioparadise with the rest of us when joining #tripleo

Feedback is welcomed!




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo] Nominate chem and matbu for tripleo-core !

[Jiří Stránský]

+1!

On 9.11.2017 09:48, Marios Andreou wrote:

Hello fellow owls,
(appologies for duplicate, forgot to add the tripleo in subject so worried
it would be missed)

I would like to nominate (and imo these are both long overdue already):

Sofer Athlan Guyot (chem)  and

Mathieu Bultel (matbu)

to tripleo-core. They have both made many many core contributions to the
upgrades & updates over the last 3 cycles touching many of the tripleo
repos (tripleo-heat-templates, tripleo-common, python-tripleoclient,
tripleo-ci, tripleo-docs and others tripleo-quickstart/extras too unless am
mistaken).

IMO their efforts and contributions are invaluable for the upgrades squad
(and beyond - see openstack overcloud config download for example) and we
will be very lucky to have them as fully voting cores. >
Please vote with +1 or -1 for either or both chem and matbu - I'll keep it
open for a week as customary,

thank you,

marios



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo]

[Jiří Stránský]

On 7.11.2017 23:29, Wesley Hayutin wrote:

Greetings,

I'd like to propose we remove the upgrade jobs that are consistently
failing from the upstream infrastructure and instead focus our efforts in
RDO Software Factory.

The jobs listed in https://review.openstack.org/#/c/518405/ are
consistently failing after being reviewed by myself and Mathieu.  I am
leaving legacy-tripleo-ci-centos-7-multinode-upgrades in place as it's
passing with an overall rate of 87%.

It doesn't make any sense to continue to tax upstream resources on failing
jobs, lets get the jobs running correctly and consistently in rdo software
factory before moving these back to our mainline CI.


I think the most useful job being removed is 
containers-multinode-upgrades, and that one i already see coming back 
green from RDO CI as green (not experimental anymore). So +1 on the removal.




Please let me know what you think of the proposal.
Thanks

The upgrade job configuration for upgrades can be found in [1]
[1]
https://github.com/rdo-infra/review.rdoproject.org-config/blob/master/jobs/tripleo-upstream.yml



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo] Nominate akrivoka for tripleo-validations core

[Jiří Stránský]

+1!

On 6.11.2017 15:32, Honza Pokorny wrote:

Hello people,

I would like to nominate Ana Krivokapić (akrivoka) for the core team for
tripleo-validations.  She has really stepped up her game on that project
in terms of helpful reviews, and great patches.

With Ana's help as a core, we can get more done, and innovate faster.

If there are no objections within a week, we'll proceed with adding Ana
to the team.

Thanks

Honza Pokorny

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo] Help needed on debugging upgrade jobs on Pike

[Jiří Stránský]

On 6.11.2017 11:17, Jiří Stránský wrote:

On 6.11.2017 10:52, Marios Andreou wrote:

On Mon, Nov 6, 2017 at 11:09 AM, Marius Cornea <mari...@redhat.com> wrote:


On Sat, Nov 4, 2017 at 2:27 AM, Emilien Macchi <emil...@redhat.com> wrote:

Since we've got promotion, we can now properly test upgrades from ocata

to pike.

It's now failing for various reasons, as you can see on:
https://review.openstack.org/#/c/500625/

I haven't filled bug yet but this is the kind of thing I see now:
http://logs.openstack.org/25/500625/20/check/legacy-tripleo-ci-centos-7-

scenario002-multinode-oooq-container-upgrades/62e7f14/
logs/undercloud/home/zuul/overcloud_upgrade_console.log.
txt.gz#_2017-11-04_00_14_17

I think this is related to https://review.openstack.org/#/c/510577/
which introduced running os-net-config during the major upgrade
composable step. In case of environments without network isolation
/etc/os-net-config/config.json doesn't exist so the os-net-config
command fails. I filed https://bugs.launchpad.net/tripleo/+bug/1730328
to keep track of it.



heh, beat me to it :) I was about to file that. Indeed from logs @ [0] you
can see the step3 ansible-playbook failing for
https://github.com/openstack/tripleo-heat-templates/blob/e463ca15fb2189fde7e7e2de136cfb2303d3171f/puppet/services/tripleo-packages.yaml#L56-L64

I had a poke at one of the other jobs too since there are apparently
multiple issues - I found a different one
for legacy-tripleo-ci-centos-7-containers-multinode-upgrades and filed
https://bugs.launchpad.net/tripleo/+bug/1730349 for that. It looks like all
the upgrade_tasks pass there but then fails on docker-puppet


I'm not sure if it's related to that ^ error in particular


Yea the backport [2] seems to have fixed that issue. The upgrade now 
completed successfully, but the job failed on validation. I've +A'd the 
backport as it gets us closer to green.



, but since we
landed deploy/upgrade scenario separation [1], the upgrade job on Pike
effectively started testing non-pacemaker to pacemaker upgrade, which
won't work. Due to a chicken-and-egg issue with landing related patches
we could not set the dependencies properly. There's a patch fixing this
issue and making the Pike upgrade pacemaker-to-pacemaker [2]. This may
not solve all the issues, but i think we need it merged to at least have
a chance at a green result.



[0]
http://logs.openstack.org/25/500625/20/check/legacy-tripleo-ci-centos-7-scenario002-multinode-oooq-container-upgrades/62e7f14/logs/subnode-2/var/log/messages.txt.gz#_Nov__4_00_13_55

[1] https://review.openstack.org/#/c/500552
[2] https://review.openstack.org/#/c/512305


thanks,

marios



I'm requesting some help from the upgrades squad, if they already saw

the failures, etc. It would be great to have the jobs passing at some
point, now the framework is in place and we had promotion.

Thanks,


--

Emilien Macchi






__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo] Help needed on debugging upgrade jobs on Pike

[Jiří Stránský]

On 6.11.2017 10:52, Marios Andreou wrote:

On Mon, Nov 6, 2017 at 11:09 AM, Marius Cornea  wrote:


On Sat, Nov 4, 2017 at 2:27 AM, Emilien Macchi  wrote:

Since we've got promotion, we can now properly test upgrades from ocata

to pike.

It's now failing for various reasons, as you can see on:
https://review.openstack.org/#/c/500625/

I haven't filled bug yet but this is the kind of thing I see now:
http://logs.openstack.org/25/500625/20/check/legacy-tripleo-ci-centos-7-

scenario002-multinode-oooq-container-upgrades/62e7f14/
logs/undercloud/home/zuul/overcloud_upgrade_console.log.
txt.gz#_2017-11-04_00_14_17

I think this is related to https://review.openstack.org/#/c/510577/
which introduced running os-net-config during the major upgrade
composable step. In case of environments without network isolation
/etc/os-net-config/config.json doesn't exist so the os-net-config
command fails. I filed https://bugs.launchpad.net/tripleo/+bug/1730328
to keep track of it.



heh, beat me to it :) I was about to file that. Indeed from logs @ [0] you
can see the step3 ansible-playbook failing for
https://github.com/openstack/tripleo-heat-templates/blob/e463ca15fb2189fde7e7e2de136cfb2303d3171f/puppet/services/tripleo-packages.yaml#L56-L64

I had a poke at one of the other jobs too since there are apparently
multiple issues - I found a different one
for legacy-tripleo-ci-centos-7-containers-multinode-upgrades and filed
https://bugs.launchpad.net/tripleo/+bug/1730349 for that. It looks like all
the upgrade_tasks pass there but then fails on docker-puppet


I'm not sure if it's related to that ^ error in particular, but since we 
landed deploy/upgrade scenario separation [1], the upgrade job on Pike 
effectively started testing non-pacemaker to pacemaker upgrade, which 
won't work. Due to a chicken-and-egg issue with landing related patches 
we could not set the dependencies properly. There's a patch fixing this 
issue and making the Pike upgrade pacemaker-to-pacemaker [2]. This may 
not solve all the issues, but i think we need it merged to at least have 
a chance at a green result.




[0]
http://logs.openstack.org/25/500625/20/check/legacy-tripleo-ci-centos-7-scenario002-multinode-oooq-container-upgrades/62e7f14/logs/subnode-2/var/log/messages.txt.gz#_Nov__4_00_13_55

[1] https://review.openstack.org/#/c/500552
[2] https://review.openstack.org/#/c/512305


thanks,

marios



I'm requesting some help from the upgrades squad, if they already saw

the failures, etc. It would be great to have the jobs passing at some
point, now the framework is in place and we had promotion.

Thanks,


--

Emilien Macchi






__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo] Repo structure for ansible-k8s-roles-* under TripleO's umbrella

[Jiří Stránský]

On 9.10.2017 11:29, Flavio Percoco wrote:

Greetings,

I've been working on something called triple-apbs (and it's respective roles) in
the last couple of months. You can find more info about this work here[0][1][2]

This work is at the point where I think it would be worth start discussing how
we want these repos to exist under the TripleO umbrella. As far as I can tell,
we have 2 options (please comment with alternatives if there are more):

1. A repo per role: Each role would have its own repo - this is the way I've
been developing it on Github. This model is closer to the ansible way of doing
things and it'll make it easier to bundle, ship, and collaborate on, individual
roles. Going this way would produce something similar to what the
openstack-ansible folks have.

2. Everything in a single repo: this would ease the import process and
integration with the rest of TripleO. It'll make the early days of this work a
bit easier but it will take us in a direction that doesn't serve one of the
goals of this work.

My preferred option is #1 because one of the goals of this work is to have
independent roles that can also be consumed standalone. In other words, I would
like to stay closer to the ansible recommended structure for roles. Some
examples[3][4]

Any thoughts? preferences?


+1 for option #1. In addition to standalone usage, it feels like a 
better match for "the container way of doing things" in that we'll be 
able to easily mix and match APB versions when necessary. (E.g. having 
problems with bleeding edge Glance APB? Just use a slightly older one 
without being compelled to downgrade the other APBs.) A parallel could 
be drawn to how openstack/puppet-* repos are managed and IMO it's been 
working well that way.


Using APBs this way also seems more "out-of-the-box ready" for APBs that 
don't originate in TripleO project, should we ever want/need to use them 
(e.g. for non-OpenStack services).


Global changes will be harder as they'll require separate commits, and 
in general it's more repos (+ RPMs) to manage, but i hope the 
aforementioned benefits outweigh this.


Jirka


Flavio

[0] http://blog.flaper87.com/deploy-mariadb-kubernetes-tripleo.html
[1] http://blog.flaper87.com/glance-keystone-mariadb-on-k8s-with-tripleo.html
[2] https://github.com/tripleo-apb
[3] https://github.com/tripleo-apb/ansible-role-k8s-mariadb
[4] https://github.com/tripleo-apb/ansible-role-k8s-glance

--
@flaper87
Flavio Percoco



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo] Unbranched repositories and testing

[Jiří Stránský]

On 5.10.2017 22:40, Alex Schultz wrote:

Hey folks,

So I wandered across the policy spec[0] for how we should be handling
unbranched repository reviews and I would like to start a broader
discussion around this topic.  We've seen it several times over the
recent history where a change in oooqe or tripleo-ci ends up affecting
either a stable branch or an additional set of jobs that were not run
on the change.  I think it's unrealistic to run every possible job
combination on every submission and it's also a giant waste of CI
resources.  I also don't necessarily agree that we should be using
depends-on to prove things are fine for a given patch for the same
reasons. That being said, we do need to minimize our risk for patches
to these repositories.

At the PTG retrospective I mentioned component design structure[1] as
something we need to be more aware of. I think this particular topic
is one of those types of things where we could benefit from evaluating
the structure and policy around these unbranched repositories to see
if we can improve it.  Is there a particular reason why we continue to
try and support deployment of (at least) 3 or 4 different versions
within a single repository?  Are we adding new features that really
shouldn't be consumed by these older versions such that perhaps it
makes sense to actually create stable branches?  Perhaps there are
some other ideas that might work?


Other folks probably have a better view of the full context here, but 
i'll chime in with my 2 cents anyway..


I think using stable branches for tripleo-quickstart-extras could be 
worth it. The content there is quite tightly coupled with the expected 
TripleO end-user workflows, which tend to evolve considerably between 
releases. Branching extras might be a good way to "match the reality" in 
that sense, and stop worrying about breaking older workflows. (Just 
recently it came up that the upgrade workflow in O is slightly updated 
to make it work in P, and will change quite a bit for Q. Minor updates 
also changed between O and P.)


I'd say that tripleo-quickstart is a different story though. It seems 
fairly release-agnostic in its focus. We may want to keep it unbranched 
(?). That probably applies even more for tripleo-ci, where ability to 
make changes which affect how TripleO does CIing in general, across 
releases, is IMO a significant feature.


Maybe branching quickstart-extras might require some code reshuffling 
between what belongs there and what belongs into quickstart itself.


(Just my 2 cents, i'm likely not among the most important stakeholders 
in this...)


Jirka



Thanks,
-Alex

[0] https://review.openstack.org/#/c/478488/
[1] http://people.redhat.com/aschultz/denver-ptg/tripleo-ptg-retro.jpg

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO] TripleO/Ansible PTG session

[Jiří Stránský]

On 22.9.2017 15:30, Jiri Tomasek wrote:

Will it be possible to send Zaqar messages at each deployment step to make
the deployment process more interactive?


If we go the way of allowing the wrapping playbook to execute per step, 
i think we could send messages to Zaqar after each step. Mistral could 
trigger the playbook with sth like `tripleo_run_steps: [1]`, then with 
`tripleo_run_steps: [2]` etc. So even though the playbook would support 
running all steps by default, Mistral could run it step-by-step.


(Currently we only send Zaqar messages before Heat stack and after Heat 
stack, is that right? [1] Or do we send & read some messages from within 
the overcloud stack deployment too?)


With logs from external playbooks (like ceph-ansible) it would be less 
comfy than it is now, or would require extra effort to improve on that. 
Currently AFAIK we don't send any Zaqar messages there anyway [2], but 
we at least publish the logs into Mistral separately from the rest of 
deployment logs. If we trigger an external playbook from Ansible, making 
it a "normal citizen" of a TripleO deployment step, then the external 
playbook logs would be reported along with logs of that whole deployment 
step. This is IMO the main weak point of having the step loop in Ansible 
(assuming its most basic form), perhaps something where we could find an 
improvement, e.g. utilizing ARA as James suggested earlier in the 
thread. (I don't know enough about ARA at this point to suggest a 
particular way of integrating it with Mistral/UI though.)



in case of driving separate
playbooks from mistral workflow, that would be absolutely possible. As it
seems we're more keen on driving everything from wrapping ansible playbook,
is it going to be possible to send Zaqar messages from ansible playbook
directly?


I did a quick search but didn't find any existing Ansible module for 
this. If we needed/wanted this for additional granularity, we might have 
to implement it.




Being able to properly monitor progress of deployment is important so it
would be good to clarify how that is going to work.

-- Jirka


[1] 
https://github.com/openstack/tripleo-common/blob/979dc308e51bb9b8e7b66b4258da1d67e50d9c2b/workbooks/deployment.yaml#L180
[2] 
https://github.com/openstack/tripleo-common/blob/979dc308e51bb9b8e7b66b4258da1d67e50d9c2b/workbooks/ceph-ansible.yaml


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO] TripleO/Ansible PTG session

[Jiří Stránský]

On 22.9.2017 13:44, Giulio Fidente wrote:

On 09/21/2017 07:53 PM, Jiří Stránský wrote:

On 21.9.2017 18:04, Marios Andreou wrote:

On Thu, Sep 21, 2017 at 3:53 PM, Jiří Stránský <ji...@redhat.com> wrote:


[...]


That way we could run the whole thing end-to-end via
ansible-playbook, or
if needed one could execute smaller bits by themselves (steps or nested
playbook runs) -- that capability is not baked in by default, but i
think
we could make it so.

Also the interface for services would be clean and simple -- it's always
the ansible tasks.

And Mistral-less use cases become easier to handle too (= undercloud
installation when Mistral isn't present yet, or development envs when
you
want to tune the playbook directly without being forced to go through
Mistral).



You don't *have* to go through mistral either way I mean you can always
just run ansible-playbook directly using the generated playbooks if
that is
what you need for dev/debug etc.




Logging becomes a bit more unwieldy in this scenario though, as for the
nested ansible-playbook execution, all output would go into a task in
the
outer playbook, which would be harder to follow and the log of the outer
playbook could be huge.

So this solution is no silver bullet, but from my current point of
view it
seems a bit less conceptually foreign than using Mistral to provide step
loop functionality to Ansible, which should be able to handle that on
its
own.



just saying using mistral to invoke ansible-playbook doesn't imply having
mistral do the looping/step control. I think it was already mentioned
that
we can/will have multiple invocations of ansible-playbook. Having the
loop
in the playbook then means organising our templates a certain way so that
there is a _single_ parent playbook which we can parameterise to then run
all or some of the steps (which as pointed above is currently the case
for
the upgrade and deployment steps playbooks).


Yup, +1 again :) However, the 1)2)3)4) approach discussed earlier in the
thread suggested to hand over the step loop control to Mistral and keep
using the Mistral workflow_tasks, which would make it impossible to have
a single parent playbook, if i understood correctly. So Mistral would be
a requirement for running all steps via a single command (impacting UC
install and developer workflow).


yes I am not sold (yet?) on the idea of ansible driving the deployment
and would like to keep some abstraction before it

the additional abstraction will make it possible for example to execute
tasks written as mistral actions (eg. python code) in between or during
any given deployment step, instead of ansible tasks only ... I guess we
could also write ansible actions in python but it's not trivial to ship
them from THT and given the project mission we have of being "openstack
on openstack" I'd also prefer writing a mistral action vs ansible

similarily, the ceph-ansible workflow runs a task to build the ansible
inventory; if we make the "external" services integration an
ansible->ansible process we'll probably need to ship from THT an heat
query (or ansible task) to be executed by the "outer" ansible to create
the inventory for the inner ansible


Yea, allowing e2e software deployment with Ansible requires converting 
the current Mistral workflow_tasks into Ansible. In terms of services 
affected by this, there's in-tree ceph-ansible [1] and we have proposed 
patches for Kubernetes and Skydive (that's what i'm aware of).




I supported the introduction of mistral as an API and would prefer to
have there more informations there versus moving it away into YACT (yet
another configuration tool)


We could mitigate this somewhat by doing what Marios and James suggested 
-- running the global playbook one step at a time when the playbook is 
executed from Mistral. It will not give Mistral 100% of the information 
when compared with the approach you suggested, but it's a bit closer...




depending on mistral for the undercloud install is also not very
different from depending on heat(-all)

I understand the ansible->ansible process addresses the "simplification"
issue we have been asked to look into; it is pretty much the only good
thing I see about it though :D


Right, it's a simpler design, which i consider important, as my hope is 
that over time it would result in some time savings, hopefully 
not just among developers, but also operators, when having to debug 
things or reason about how TripleO works.


Btw the points i didn't react to -- i mostly agree there :P. There are 
tradeoffs involved in both variants and it's not a clear-as-day choice 
for me either.


Thanks :)

Jirka

[1] 
https://github.com/openstack/tripleo-common/blob/master/workbooks/ceph-ansible.yaml


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:un

Re: [openstack-dev] [TripleO] TripleO/Ansible PTG session

[Jiří Stránský]

On 21.9.2017 18:04, Marios Andreou wrote:

On Thu, Sep 21, 2017 at 3:53 PM, Jiří Stránský <ji...@redhat.com> wrote:


On 21.9.2017 12:31, Giulio Fidente wrote:


On 09/20/2017 07:36 PM, James Slagle wrote:


On Tue, Sep 19, 2017 at 8:37 AM, Giulio Fidente <gfide...@redhat.com>
wrote:


On 09/18/2017 05:37 PM, James Slagle wrote:


- The entire sequence and flow is driven via Mistral on the Undercloud
by default. This preserves the API layer and provides a clean reusable
interface for the CLI and GUI.



I think it's worth saying that we want to move the deployment steps out
of heat and in ansible, not in mistral so that mistral will run the
workflow only once and let ansible go through the steps

I think having the steps in mistral would be a nice option to be able to
rerun easily a particular deployment step from the GUI, versus having
them in ansible which is instead a better option for CLI users ... but
it looks like having them in ansible is the only option which permits us
to reuse the same code to deploy an undercloud because having the steps
in mistral would require the undercloud installation itself to depend on
mistral which we don't want to

James, Dan, please comment on the above if I am wrong



That's correct. We don't want to require Mistral to install the
Undercloud. However, I don't think that necessarily means it has to be
a single call to ansible-playbook. We could have multiple invocations
of ansible-playbook. Both Mistral and CLI code for installing the
undercloud could handle that easily.

You wouldn't be able to interleave an external playbook among the
deploy steps however. That would have to be done under a single call
to ansible-playbook (at least how that is written now). We could
however have hooks that could serve as integration points to call
external playbooks after each step.



the benefits of driving the steps from mistral are that then we could
also interleave the deployment steps and we won't need the
ansible-playbook hook for the "external" services:

1) collect the ansible tasks *and* the workflow_tasks (per step) from heat

2) launch the stepN deployment workflow (ansible-playbook)

3) execute any workflow_task defined for stepN (like ceph-ansible
playbook)

4) repeat 2 and 3 for stepN+1

I think this would also provide a nice interface for the UI ... but then
we'd need mistral to be able to deploy the undercloud




Why not launch the _single_  deploy_steps playbook (so we have
when/conditionals with step numbers), passing in the step you want to have
executed (we can pass this in as a parameter to the mistral workflow and
pass through to the ansible-playbook invocation?), otherwise execute all
the steps.


+1 that's the sort of thing i meant by "it's not baked in by default but 
we could make it so". We could even give it a list of steps like 
`tripleo_run_steps: [4, 5, 6]`.



In either case whether it should be ansible handling the loop
based on a passed in parameter.
'Ansible-native' looping is currently the
case for the current deploy_steps_playbook here
https://github.com/openstack/tripleo-heat-templates/blob/259cf512b3b7e3539105cdb52421e3239701ef73/common/deploy-steps.j2#L335
- can we not work parameterise that playbook so that we either do loop with
the variable, or just step X?

Then for the upgrade workflow it is as above but  1.5 first launch the
upgrade_tasks_playbook  then the deploy_steps playbook for all the steps
(consider this
https://review.openstack.org/#/c/505624/3/scripts/upgrade-non-controller.sh@162
- download and run the playbooks for non-controllers in O->P upgrade
pointing this out to illustrate the workflow I have in mind). So I don't
see why we can't have mistral invoking ansible-playbook and taking
parameters like which playbook, which step etc.




Alternatively we could do the main step loop in Ansible directly, and have
the tasks do whatever they need to get the particular service deployed,
from  to launching a nested ansible-playbook run if that's what it takes.




I think you can do both, I mean mistral invoking ansible-playbook and still
let ansible handle the steps with a loop.


+1 exactly. FWIW i'm totally on board with wrapping everything in 
Mistral on the outermost level, as that's required for UI. I'm just not 
keen on having Mistral enter the process in between each step and drive 
the step loop.



In fact that is what the current
upgrade_steps_playbook does here
https://github.com/openstack/tripleo-heat-templates/blob/259cf512b3b7e3539105cdb52421e3239701ef73/common/deploy-steps.j2#L363-L365
with a loop variable 'step'. The upgrade_tasks have their 'tags: stepX'
converted to 'when: step == X' in the client here
https://github.com/openstack/python-tripleoclient/blob/4d342826d6c3db38ee88dccc92363b655b1161a5/tripleoclient/v1/overcloud_config.py#L63
- we must come up with a better solution than that; ultimately we can just
update the existing upgrade_tasks to have 'when' and the m

Re: [openstack-dev] [TripleO] TripleO/Ansible PTG session

[Jiří Stránský]

On 21.9.2017 12:31, Giulio Fidente wrote:

On 09/20/2017 07:36 PM, James Slagle wrote:

On Tue, Sep 19, 2017 at 8:37 AM, Giulio Fidente  wrote:

On 09/18/2017 05:37 PM, James Slagle wrote:

- The entire sequence and flow is driven via Mistral on the Undercloud
by default. This preserves the API layer and provides a clean reusable
interface for the CLI and GUI.


I think it's worth saying that we want to move the deployment steps out
of heat and in ansible, not in mistral so that mistral will run the
workflow only once and let ansible go through the steps

I think having the steps in mistral would be a nice option to be able to
rerun easily a particular deployment step from the GUI, versus having
them in ansible which is instead a better option for CLI users ... but
it looks like having them in ansible is the only option which permits us
to reuse the same code to deploy an undercloud because having the steps
in mistral would require the undercloud installation itself to depend on
mistral which we don't want to

James, Dan, please comment on the above if I am wrong


That's correct. We don't want to require Mistral to install the
Undercloud. However, I don't think that necessarily means it has to be
a single call to ansible-playbook. We could have multiple invocations
of ansible-playbook. Both Mistral and CLI code for installing the
undercloud could handle that easily.

You wouldn't be able to interleave an external playbook among the
deploy steps however. That would have to be done under a single call
to ansible-playbook (at least how that is written now). We could
however have hooks that could serve as integration points to call
external playbooks after each step.


the benefits of driving the steps from mistral are that then we could
also interleave the deployment steps and we won't need the
ansible-playbook hook for the "external" services:

1) collect the ansible tasks *and* the workflow_tasks (per step) from heat

2) launch the stepN deployment workflow (ansible-playbook)

3) execute any workflow_task defined for stepN (like ceph-ansible playbook)

4) repeat 2 and 3 for stepN+1

I think this would also provide a nice interface for the UI ... but then
we'd need mistral to be able to deploy the undercloud



Alternatively we could do the main step loop in Ansible directly, and 
have the tasks do whatever they need to get the particular service 
deployed, from  to launching a nested ansible-playbook run if that's 
what it takes.


That way we could run the whole thing end-to-end via ansible-playbook, 
or if needed one could execute smaller bits by themselves (steps or 
nested playbook runs) -- that capability is not baked in by default, but 
i think we could make it so.


Also the interface for services would be clean and simple -- it's always 
the ansible tasks.


And Mistral-less use cases become easier to handle too (= undercloud 
installation when Mistral isn't present yet, or development envs when 
you want to tune the playbook directly without being forced to go 
through Mistral).


Logging becomes a bit more unwieldy in this scenario though, as for the 
nested ansible-playbook execution, all output would go into a task in 
the outer playbook, which would be harder to follow and the log of the 
outer playbook could be huge.


So this solution is no silver bullet, but from my current point of view 
it seems a bit less conceptually foreign than using Mistral to provide 
step loop functionality to Ansible, which should be able to handle that 
on its own.



- It would still be possible to run ansible-playbook directly for
various use cases (dev/test/POC/demos). This preserves the quick
iteration via Ansible that is often desired.

- The remaining SoftwareDeployment resources in tripleo-heat-templates
need to be supported by config download so that the entire
configuration can be driven with Ansible, not just the deployment
steps. The success criteria for this point would be to illustrate
using an image that does not contain a running os-collect-config.

- The ceph-ansible implementation done in Pike could be reworked to
use this model. "config download" could generate playbooks that have
hooks for calling external playbooks, or those hooks could be
represented in the templates directly. The result would be the same
either way though in that Heat would no longer be triggering a
separate Mistral workflow just for ceph-ansible.


I'd say for ceph-ansible, kubernetes and in general anything else which
needs to run with a standard playbook installed on the undercloud and
not one generated via the heat templates... these "external" services
usually require the inventory file to be in different format, to
describe the hosts to use on a per-service basis, not per-role (and I
mean tripleo roles here, not ansible roles obviously)

About that, we discussed a more long term vision where the playbooks
(static data) needd to describe how to deploy/upgrade a given service is
in a separate repo 

Re: [openstack-dev] [tripleo] Install Kubernetes in the overcloud using TripleO

[Jiří Stránský]

On 20.9.2017 10:15, Bogdan Dobrelya wrote:

On 08.06.2017 18:36, Flavio Percoco wrote:

Hey y'all,

Just wanted to give an updated on the work around tripleo+kubernetes.
This is
still far in the future but as we move tripleo to containers using
docker-cmd,
we're also working on the final goal, which is to have it run these
containers
on kubernetes.

One of the first steps is to have TripleO install Kubernetes in the
overcloud
nodes and I've moved forward with this work:

https://review.openstack.org/#/c/471759/

The patch depends on the `ceph-ansible` work and it uses the
mistral-ansible
action to deploy kubernetes by leveraging kargo. As it is, the patch
doesn't
quite work as it requires some files to be in some places (ssh keys) and a
couple of other things. None of these "things" are blockers as in they
can be
solved by just sending some patches here and there.

I thought I'd send this out as an update and to request some early
feedback on
the direction of this patch. The patch, of course, works in my local
environment
;)


Note that Kubespray (former Kargo) now supports the kubeadm tool
natively [0]. This speeds up a cluster bootstrapping from an average
25-30 min to a 9 or so. I believe this makes Kubespray a viable option
for upstream development of OpenStack overclouds managed by K8s.
Especially, bearing in mind the #deployment-time effort and all the hard
work work done by tripleo and infra teams in order to shorten the CI
jobs time.


I tried deploying with kubeadm_enable yesterday and no luck yet on 
CentOS, but i do want to get back to this as the speed up sounds 
promising :)


AIO kubernetes deployment the non-kubeadm way seemed to work fine 
(Flavio's patch above with a workaround for [2] and a small Kubespray 
fix [3]).


Jirka



By the way, here is a package review [1] for adding a kubespray-ansible
library, just ansible roles and playbooks, to RDO. I'd appreciate some
help with moving this forward, like choosing another place to host the
package, it got stuck a little bit.

[0] https://github.com/kubernetes-incubator/kubespray/issues/553
[1] https://bugzilla.redhat.com/show_bug.cgi?id=1482524

[2] https://bugs.launchpad.net/mistral/+bug/1718384
[3] https://github.com/kubernetes-incubator/kubespray/pull/1677

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo] Gate is broken - Do not approve any patch until further notice

[Jiří Stránský]

On 30.8.2017 06:54, Emilien Macchi wrote:

On Tue, Aug 29, 2017 at 4:17 PM, Emilien Macchi  wrote:

We are currently dealing with 4 issues and until they are fix, please
do not approve any patch. We want to keep the gate clear to merge the
fixes for the 4 problems first.

1) devstack-gate broke us because we use it as a library (bad)
https://bugs.launchpad.net/tripleo/+bug/1713868

2) https://review.openstack.org/#/c/474578/ broke us and we're
reverting it https://bugs.launchpad.net/tripleo/+bug/1713832

3) We shouldn't build images on multinode jobs
https://bugs.launchpad.net/tripleo/+bug/1713167

4) We should use pip instead of git for delorean
https://bugs.launchpad.net/tripleo/+bug/1708832


Until further notice from Alex or myself, please do not approve any patch.


The 4 problems have been mitigated.
You can now proceed to normal review.

Please do not recheck a patch without an elastic-recheck comment, we
need to track all issues related to CI from now.
Paul Belanger has been doing extremely useful work to help us, now
let's use elastic-recheck more and stop blind rechecks.
All known issues are in http://status.openstack.org/elastic-recheck/
If one is missing, you're welcome to contribute by sending a patch to
elastic-recheck. Example with https://review.openstack.org/#/c/498954/


Posted DLRN build failure query [1]. I used the Kibana interface [2] to 
test-drive the query.


I wanted to tackle other bugs but it seems we don't have enough info in 
console.html. I wonder if it's realistic to start pulling some logs 
maybe from undercloud/home/jenkins dir into logstash? That's where OOOQ 
puts the most of its more detailed output, so having that might allow us 
to produce more specific queries.


Thanks,

Jirka

[1] https://review.openstack.org/499532
[2] http://logstash.openstack.org



I've restored all patches that were killed from the gate and did
recheck already, hopefully we can get some merges and finish this
release.

Thanks Paul and all Infra for their consistent help!




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo] Pacemaker + containers CI

[Jiří Stránský]

On 29.8.2017 17:12, Emilien Macchi wrote:

On Tue, Aug 29, 2017 at 2:14 AM, Jiří Stránský <ji...@redhat.com> wrote:
[...]

the CI for containerized deployments with Pacemaker is close! In fact, it
works [1][2] (but there are pending changes to merge).


Really good news, thanks for the update!


The way it's proposed in gerrit currently is to switch the
centos-7-containers-multinode job (featureset010) to deploy with Pacemaker.
What do you think about making this switch as a first step? [...]


I'm ok with the idea


No -1s yet, so i removed WIP status of [4].


as long as
gate-tripleo-ci-centos-7-containers-multinode-upgrades-nv keep working
fine.


That's a different featureset so we can control it independently from 
the basic deployment job. It might be good to switch this one to 
Pacemaker too, if we can solve the current timeout issues and perhaps 
have some spare wall time.


Non-pacemaker containers are still CI'd by OVB job, so the upgrade job 
(currently still non-Pacemaker) shouldn't be more vulnerable even if we 
switch the multinode job to Pacemaker.



Deploying Pacemaker on a single node environment is not optimal but
already cover a bunch of code which is good.


Later it would be nice to get a proper clustering test with 3 controllers.
Should we try and switch the centos-7-ovb-ha-oooq job to deploy containers
on master and stable/pike? (Probably by adding a new job that only runs on
master + Pike, and making the old ovb-ha-oooq only run upto Ocata, to keep
the OVB capacity demands unchanged?) I'd be +1 on that since containers are
the intended way of deploying Pike and beyond. WDYT?


It's actually a good start to our discussion at the PTG:
https://etherpad.openstack.org/p/tripleo-ptg-queens-ci-related-topics
(we have a session on Wednesday morning about CI topics, please make
sure you can join!)

I think in Queens, we'll run container-only jobs, even for OVB.
That said, I think OVB coverage in Queens will be very useful to try
HA with 3 controllers (containerized) and the baremetal services
coverage will only run on Pike, Ocata and Newton.

That way, we would have:

Queens:
- multinode jobs covering basic HA scenario, single node but still
useful to test a good part of the code
- OVB jobs covering production environment and hopefully spot issues
we wouldn't see with multinode jobs

Pike, Ocata, Newton:
no change on OVB job

(note it's a proposal, not a statement)


Yea focusing the CI changes towards containerized mainly on Queens+ 
could be fine too. The frequency of patches going into stable/pike will 
be dropping as it gains stability, so time spent on CI enhancements 
might indeed be better focused on Queens+. We can always adjust if that 
doesn't prove to be the case.




[...]

[3] https://review.openstack.org/498474


approved

[...]

Thanks,




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo] Pacemaker + containers CI

[Jiří Stránský]

On 29.8.2017 14:42, Giulio Fidente wrote:

On 08/29/2017 02:33 PM, Jiří Stránský wrote:

A bit of context: Currently our only upgrade check job is non-OVB -
containers-multinode-upgrades-nv. As of late we started hitting
timeouts, and the job only does mixed-version deploy + 1 node AIO
overcloud upgrade (just the main step). It doesn't do undercloud
upgrade, nor compute upgrade, nor converge, and it still times out...
It's a bit difficult to find things to cut off here. :D We could look
into speeding things up (e.g. try to reintroduce selective container
image upload etc.) but i think we might also be approaching the
"natural" deploy+upgrade limits. We might need to bump up the timeouts
if we want to test more things. Though it's not only about capacity of
HW, it could also get unwieldy for devs if we keep increasing the
feedback time from CI, so we're kinda in a tough spot with upgrade CI...


agreed which goes back to "nobody looks at the periodic jobs" but
periodic job seems the answer?



Yea that might be the best solution :)

J.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo] Pacemaker + containers CI

[Jiří Stránský]

On 29.8.2017 13:22, Giulio Fidente wrote:

On 08/29/2017 11:14 AM, Jiří Stránský wrote:

Hi owls,

the CI for containerized deployments with Pacemaker is close! In fact,
it works [1][2] (but there are pending changes to merge).


cool :D

I also spotted this which we need for ceph
https://review.openstack.org/#/c/498356/

but I am not sure if we want to enable ceph in this job as we have it
already in a couple of scenarios, more below ...


+1 on keeping it in scenarios if that covers our needs.




The way it's proposed in gerrit currently is to switch the
centos-7-containers-multinode job (featureset010) to deploy with
Pacemaker. What do you think about making this switch as a first step?
(The OVB job is an option too, but that one is considerably closer to
timeouts already, so it may be better left as is.)


+1 on switching the existing job


Later it would be nice to get a proper clustering test with 3
controllers. Should we try and switch the centos-7-ovb-ha-oooq job to
deploy containers on master and stable/pike? (Probably by adding a new
job that only runs on master + Pike, and making the old ovb-ha-oooq only
run upto Ocata, to keep the OVB capacity demands unchanged?) I'd be +1
on that since containers are the intended way of deploying Pike and
beyond. WDYT?


switching OVB to containers from pike seems fine because that's the
indended way as you pointed, yet I would like to enable ceph in the
upgrade job, and it requires multiple MON instances (multiple controllers)

would it make any sense to deploy the pacemaker / ceph combination using
multiple controllers in the upgrade job and drop the standard ovb job
(which doesn't do upgrade) or use it for other purposes?


It makes sense feature-wise to test upgrade with Ceph, i'd say it's a 
pretty common and important use case.


However i'm not sure how can we achieve it time-wise in CI. Is it 
possible to estimate how much time might the Ceph upgrade add?


A bit of context: Currently our only upgrade check job is non-OVB - 
containers-multinode-upgrades-nv. As of late we started hitting 
timeouts, and the job only does mixed-version deploy + 1 node AIO 
overcloud upgrade (just the main step). It doesn't do undercloud 
upgrade, nor compute upgrade, nor converge, and it still times out... 
It's a bit difficult to find things to cut off here. :D We could look 
into speeding things up (e.g. try to reintroduce selective container 
image upload etc.) but i think we might also be approaching the 
"natural" deploy+upgrade limits. We might need to bump up the timeouts 
if we want to test more things. Though it's not only about capacity of 
HW, it could also get unwieldy for devs if we keep increasing the 
feedback time from CI, so we're kinda in a tough spot with upgrade CI...


Jirka

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [tripleo] Pacemaker + containers CI

[Jiří Stránský]

Hi owls,

the CI for containerized deployments with Pacemaker is close! In fact, 
it works [1][2] (but there are pending changes to merge).


The way it's proposed in gerrit currently is to switch the 
centos-7-containers-multinode job (featureset010) to deploy with 
Pacemaker. What do you think about making this switch as a first step? 
(The OVB job is an option too, but that one is considerably closer to 
timeouts already, so it may be better left as is.)


Later it would be nice to get a proper clustering test with 3 
controllers. Should we try and switch the centos-7-ovb-ha-oooq job to 
deploy containers on master and stable/pike? (Probably by adding a new 
job that only runs on master + Pike, and making the old ovb-ha-oooq only 
run upto Ocata, to keep the OVB capacity demands unchanged?) I'd be +1 
on that since containers are the intended way of deploying Pike and 
beyond. WDYT?


Have a good day,

Jirka

P.S. You can deploy containerized with pacemaker using OOOQ by setting 
both `containerized_overcloud` and `enable_pacemaker` to true. Thanks to 
Wes for collaboration on this.


P.P.S. The remaining patches are [3] and maybe [4] if we're ok with 
switching centos-7-containers-multinode.



[1] 
http://logs.openstack.org/24/471724/5/check/gate-tripleo-ci-centos-7-containers-multinode/6330e5e/logs/subnode-2/var/log/pacemaker/bundles/


[2] 
http://logs.openstack.org/24/471724/5/check/gate-tripleo-ci-centos-7-containers-multinode/6330e5e/logs/subnode-2/var/log/extra/docker/containers/


[3] https://review.openstack.org/498474
[4] https://review.openstack.org/471724

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo] Adding new roles after upgrade is broken.

[Jiří Stránský]

On 18.8.2017 17:25, Marios Andreou wrote:

On Fri, Aug 18, 2017 at 4:22 PM, Jiří Stránský <ji...@redhat.com> wrote:


On 18.8.2017 13:18, Sofer Athlan-Guyot wrote:


Hi,

We may have missing packages when the user is adding a new role to its
roles_data file and the base image is coming from previous version.

The workflow would be this one:
   - install newton
   - upgrade to ocata
   - add collectd to roles_data and redeploy the stack

For instance if one is adding
OS::TripleO::Services::Collectdservices::collectd in an ocata env coming
from an upgraded newton env, he/she won't have the necessary packages
(for instance collectd-disk).  The puppet manifest will fail has the
package is missing and puppet doesn't install package.  The upgrade
task[1] is useless as the new role wasn't added during the upgrade but
after.



Right, but the package could be added during the upgrade. The
upgrade_tasks could/should make the set of installed overcloud RPMs on par
with the overcloud-full image of the respective release, ideally. So you'd
have collectd RPMs installed always, both on freshly deployed and upgraded
envs, regardless if you actually use collectd or not. We already did some
package installs/uninstalls as part of upgrades and updates, but probably
didn't have 100% coverage.



yeah +1 to this except where would those upgrade_tasks go? Taking the given
example, upgrade_tasks in collectd-disk.yaml wont be executed because
during the upgrade, the operator didn't have that enabled as a service (it
is default off, and new so they couldn't deploy it before).

So we may have to use some 'central place' (this is what Sofer was
advocating earlier on irc) like in the tripleo-packages.yaml and have tasks
there.


Yes exactly, tripleo-packages.yaml is the right place IMO.


The problem then becomes however that we don't _know_ which services
we need to download packages for? Oh.. no you're saying we can use the
current release package list (e.g. do you mean from something in
https://github.com/openstack/tripleo-puppet-elements pkg-maps? ).


Yea i meant diff between e.g. Newton and Ocata t-p-e:

https://github.com/openstack/tripleo-puppet-elements/compare/stable/newton...stable/ocata

Or we could download overcloud-full.qcow2 for both Newton and Ocata, and 
inspect them using libguestfs-tools for the list of installed RPMs, 
which could give us the exact diff. That might be the easiest way how to 
do this perhaps?



Have a good day folks,

Jirka







I don't see any easy way to solve this.  Basically we need a way to keep
in sync base image between release without using the upgrade_tasks,
maybe in the tripleo-package one ?



Given that released code is affected, we may treat it as a bug that
requires a minor update, and in addition to upgrade_tasks, we can add all
the necessary package installs into minor update code (yum_update.sh) too.
Again this shouldn't depend on what services are actually enabled, just
unconditionally sync with latest content of overcloud-full image of the
respective release.

I guess the time consuming part will be preparing the envs that will allow
comparing a fresh deploy vs. an upgraded one to get the `rpm -qa | sort`
difference. Or we could try a shortcut and see what changes went into
tripleo-puppet-elements in each release.



yeah based on what you said, am thinking
https://github.com/openstack/tripleo-puppet-elements/blob/master/elements/overcloud-controller/pkg-map
for example , or some parsing of the combined element pkg maps... :/ still
likely need some tooling to do that though

thanks, marios





This shouldn't be a problem with container, but everything before pike
is affected.



Indeed. There will still be some basic baremetal host content management
as long as we're not using Atomic, but the room for potential problems will
be much smaller.

Jirka



Originially seen there[2]

[1] https://github.com/openstack/tripleo-heat-templates/blob/sta
ble/ocata/puppet/services/metrics/collectd.yaml#L130..L134
[2] https://bugzilla.redhat.com/show_bug.cgi?id=1455065




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev





__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo] Adding new roles after upgrade is broken.

[Jiří Stránský]

On 18.8.2017 13:18, Sofer Athlan-Guyot wrote:

Hi,

We may have missing packages when the user is adding a new role to its
roles_data file and the base image is coming from previous version.

The workflow would be this one:
  - install newton
  - upgrade to ocata
  - add collectd to roles_data and redeploy the stack

For instance if one is adding
OS::TripleO::Services::Collectdservices::collectd in an ocata env coming
from an upgraded newton env, he/she won't have the necessary packages
(for instance collectd-disk).  The puppet manifest will fail has the
package is missing and puppet doesn't install package.  The upgrade
task[1] is useless as the new role wasn't added during the upgrade but
after.


Right, but the package could be added during the upgrade. The 
upgrade_tasks could/should make the set of installed overcloud RPMs on 
par with the overcloud-full image of the respective release, ideally. So 
you'd have collectd RPMs installed always, both on freshly deployed and 
upgraded envs, regardless if you actually use collectd or not. We 
already did some package installs/uninstalls as part of upgrades and 
updates, but probably didn't have 100% coverage.




I don't see any easy way to solve this.  Basically we need a way to keep
in sync base image between release without using the upgrade_tasks,
maybe in the tripleo-package one ?


Given that released code is affected, we may treat it as a bug that 
requires a minor update, and in addition to upgrade_tasks, we can add 
all the necessary package installs into minor update code 
(yum_update.sh) too. Again this shouldn't depend on what services are 
actually enabled, just unconditionally sync with latest content of 
overcloud-full image of the respective release.


I guess the time consuming part will be preparing the envs that will 
allow comparing a fresh deploy vs. an upgraded one to get the `rpm -qa | 
sort` difference. Or we could try a shortcut and see what changes went 
into tripleo-puppet-elements in each release.




This shouldn't be a problem with container, but everything before pike
is affected.


Indeed. There will still be some basic baremetal host content management 
as long as we're not using Atomic, but the room for potential problems 
will be much smaller.


Jirka



Originially seen there[2]

[1] 
https://github.com/openstack/tripleo-heat-templates/blob/stable/ocata/puppet/services/metrics/collectd.yaml#L130..L134
[2] https://bugzilla.redhat.com/show_bug.cgi?id=1455065




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo] critical situation with CI / upgrade jobs

[Jiří Stránský]

On 17.8.2017 00:47, Emilien Macchi wrote:

Here's an update on the situation.

On Tue, Aug 15, 2017 at 6:33 PM, Emilien Macchi  wrote:

Problem #1: Upgrade jobs timeout from Newton to Ocata
https://bugs.launchpad.net/tripleo/+bug/1702955

[...]

- revert distgit patch in RDO: https://review.rdoproject.org/r/8575
- push https://review.openstack.org/#/c/494334/ as a temporary solution
- we need https://review.openstack.org/#/c/489874/ landed ASAP.
- once https://review.openstack.org/#/c/489874/ is landed, we need to
revert https://review.openstack.org/#/c/494334 ASAP.

We still need some help to find out why upgrade jobs timeout so much
in stable/ocata.


Problem #2: from Ocata to Pike (containerized) missing container upload step
https://bugs.launchpad.net/tripleo/+bug/1710938
Wes has a patch (thanks!) that is currently in the gate:
https://review.openstack.org/#/c/493972

[...]

The patch worked and helped! We've got a successful job running today:
http://logs.openstack.org/00/461000/32/check/gate-tripleo-ci-centos-7-containers-multinode-upgrades-nv/2f13627/console.html#_2017-08-16_01_31_32_009061

We're now pushing to the next step: testing the upgrade with pingtest.
See https://review.openstack.org/#/c/494268/ and the Depends-On: on
https://review.openstack.org/#/c/461000/.

If pingtest proves to work, it would be a good news and prove that we
have a basic workflow in place on which we can iterate.

The next iterations afterward would be to work on the 4 scenarios that
are also going to be upgrades from Ocata to pike (001 to 004).
For that, we'll need Problem #1 and #2 resolved before we want to make
any progress here, to not hit the same issues that before.


Problem #3: from Ocata to Pike: all container images are
uploaded/specified, even for services not deployed
https://bugs.launchpad.net/tripleo/+bug/1710992
The CI jobs are timeouting during the upgrade process because
downloading + uploading _all_ containers in local cache takes more
than 20 minutes.
So this is where we are now, upgrade jobs timeout on that. Steve Baker
is currently looking at it but we'll probably offer some help.


Steve is still working on it: https://review.openstack.org/#/c/448328/
Steve, if you need any help (reviewing or coding) - please let us
know, as we consider this thing important to have and probably good to
have in Pike.


Independent, but related issue is that the job doesn't make use of 
CI-local registry mirrors. I seem to recall we already had mirror usage 
implemented at some point, but we must have lost it somehow. Fix is here:


https://review.openstack.org/#/c/494525/

Jirka



Thanks,




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo] Proposing Bogdan Dobrelya core on TripleO / Containers

[Jiří Stránský]

On 21.7.2017 16:55, Emilien Macchi wrote:

Hi,

Bogdan (bogdando on IRC) has been very active in Containerization of
TripleO and his quality of review has increased over time.
I would like to give him core permissions on container work in TripleO.
Any feedback is welcome as usual, we'll vote as a team.

Thanks,



+1

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO] breaking changes, new container image parameter formats

[Jiří Stránský]

On 25.7.2017 04:40, Michał Jastrzębski wrote:

...
DockerInsecureRegistryAddress: 172.19.0.2:8787/tripleoupstream
DockerKeystoneImage: 172.19.0.2:8787/tripleoupstream/centos-binary-
keystone:latest
...


That's strange construction, are you sure guys that you don't want to
separate address:port from namespace? (tripleoupstream here).

Say you'd like to setup docker to point to insecure registry (add
--insecure-registry do systemd conf), that will take addr:port not
whole thing.


Thanks Michał, i think it was just a copy-paste error in the e-mail and 
in reality the parameter value should indeed look like:


DockerInsecureRegistryAddress: 172.19.0.2:8787

I posted a tripleo-docs patch which documents the changes discussed in 
this thread:


https://review.openstack.org/486635


Thanks

Jirka

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO] breaking changes, new container image parameter formats

[Jiří Stránský]

On 19.7.2017 14:41, Dan Prince wrote:

I wanted to give a quick heads up on some breaking changes that started
landing last week with regards to how container images are specified
with Heat parameters in TripleO. There are a few patches associated
with converting over to the new changes but the primary patches are
listed below here [1] and here [2].

Here are a few examples where I'm using a local (insecure) docker
registry on 172.19.0.2.

The old parameters were:

   
   DockerNamespaceIsRegistry: true
   DockerNamespace: 172.19.0.2:8787/tripleoupstream
   DockerKeystoneImage: centos-binary-keystone:latest
   ...

The new parameters simplify things quite a bit so that each
Docker*Image parameter contains the *entire* URL required to pull the
docker image. It ends up looking something like this:

   ...
   DockerInsecureRegistryAddress: 172.19.0.2:8787/tripleoupstream
   DockerKeystoneImage: 172.19.0.2:8787/tripleoupstream/centos-binary-
keystone:latest
   ...

The benefit of the new format is that it makes it possible to pull
images from multiple registries without first staging them to a local
docker registry. Also, we've removed the 'tripleoupstream' default
container names and now require them to be specified. Removing the
default should make it much more explicit that the end user has
specified container image names correctly and doesn't accidentally use
'tripleoupstream' by accident because one of the container image
parameters didn't get specified.


Additional info based on #tripleo discussion: To keep using the values 
that were the defaults, you need to add `-e 
$THT_PATH/environments/docker-centos-tripleoupstream.yaml` [3] to the 
`openstack overcloud deploy` command.



Finally the simplification of the
DockerInsecureRegistryAddress parameter into a single setting makes
things more clear to the end user as well.

A new python-tripleoclient command makes it possible to generate a
custom heat environment with defaults for your environment and
registry. For the examples above I can run 'overcloud container image
prepare' to generate a custom heat environment like this:

openstack overcloud container image prepare --
namespace=172.19.0.2:8787/tripleoupstream --env-
file=$HOME/containers.yaml

We choose not to implement backwards compatibility with the old image
formats as almost all of the Heat parameters here are net new in Pike
and as such have not yet been released yet. The changes here should
make it much easier to manage containers and work with other community
docker registries like RDO, etc.

[1] http://git.openstack.org/cgit/openstack/tripleo-heat-templates/comm
it/?id=e76d84f784d27a7a2d9e5f3a8b019f8254cb4d6c
[2] https://review.openstack.org/#/c/479398/17
[3] 
https://github.com/openstack/tripleo-heat-templates/blob/5cbcc8377c49e395dc1d02a976d9b4a94253f5ca/environments/docker-centos-tripleoupstream.yaml


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO] Let's use Ansible to deploy OpenStack services on Kubernetes

[Jiří Stránský]

On 14.7.2017 23:00, Ben Nemec wrote:



On 07/14/2017 11:43 AM, Joshua Harlow wrote:

Out of curiosity, since I keep on hearing/reading all the tripleo
discussions on how tripleo folks are apparently thinking/doing?
redesigning the whole thing to use ansible + mistral + heat, or ansible
+ kubernetes or ansible + mistral + heat + ansible (a second time!) or ...

Seeing all those kinds of questions and suggestions around what should
be used and why and how (and even this thread) makes me really wonder
who actually uses tripleo and can afford/understand such kinds of changes?

Does anyone?

If there are  is there going to be an upgrade
path for there existing 'cloud/s' to whatever this solution is?

What operator(s) has the ability to do such a massive shift at this
point in time? Who are these 'mystical' operators?

All this has really peaked my curiosity because I am personally trying
to do that shift (not exactly the same solution...) and I know it is a
massive undertaking (that will take quite a while to get right) even for
a simple operator with limited needs out of openstack (ie godaddy); so I
don't really understand how the generic solution for all existing
tripleo operators can even work...


This is a valid point.  Up until now the answer has been that we
abstracted most of the ugliness of major changes behind either Heat or
tripleoclient.  If we end up essentially dropping those two in favor of
some other method of driving deployments it's going to be a lot harder
to migrate.  And I could be wrong, but I'm pretty sure it _is_ important
to our users to have an in-place upgrade path (see the first bullet
point in [1]).

New, shiny technology is great and all, but we do need to remember that
we have a lot of users out there already depending on the old,
not-so-shiny bits too.  They're not going to be happy if we leave them
hanging.


Exactly. Reuse is nice to have, while some sort of an upgrade path is a 
must have. We should be aware of this when selecting tools for Kubernetes.


Jirka



1: http://lists.openstack.org/pipermail/openstack-dev/2017-June/119063.html



Flavio Percoco wrote:


Greetings,

As some of you know, I've been working on the second phase of TripleO's
containerization effort. This phase if about migrating the docker based
deployment onto Kubernetes.

These phase requires work on several areas: Kubernetes deployment,
OpenStack
deployment on Kubernetes, configuration management, etc. While I've been
diving
into all of these areas, this email is about the second point, OpenStack
deployment on Kubernetes.

There are several tools we could use for this task. kolla-kubernetes,
openstack-helm, ansible roles, among others. I've looked into these
tools and
I've come to the conclusion that TripleO would be better of by having
ansible
roles that would allow for deploying OpenStack services on Kubernetes.

The existing solutions in the OpenStack community require using Helm.
While I
like Helm and both, kolla-kubernetes and openstack-helm OpenStack
projects, I
believe using any of them would add an extra layer of complexity to
TripleO,
which is something the team has been fighting for years years -
especially now
that the snowball is being chopped off.

Adopting any of the existing projects in the OpenStack communty would
require
TripleO to also write the logic to manage those projects. For example,
in the
case of openstack-helm, the TripleO team would have to write either
ansible
roles or heat templates to manage - install, remove, upgrade - the
charts (I'm
happy to discuss this point further but I'm keepping it at a
high-level on
purpose for the sake of not writing a 10k-words-long email).

James Slagle sent an email[0], a couple of days ago, to form TripleO
plans
around ansible. One take-away from this thread is that TripleO is
adopting
ansible more and more, which is great and it fits perfectly with the
conclusion
I reached.

Now, what this work means is that we would have to write an ansible role
for
each service that will deploy the service on a Kubernetes cluster.
Ideally these
roles will also generate the configuration files (removing the need of
puppet
entirely) and they would manage the lifecycle. The roles would be
isolated and
this will reduce the need of TripleO Heat templates. Doing this would
give
TripleO full control on the deployment process too.

In addition, we could also write Ansible Playbook Bundles to contain
these roles
and run them using the existing docker-cmd implementation that is coming
out in
Pike (you can find a PoC/example of this in this repo[1]).

Now, I do realize the amount of work this implies and that this is my
opinion/conclusion. I'm sending this email out to kick-off the
discussion and
gather thoughts and opinions from the rest of the community.

Finally, what I really like about writing pure ansible roles is that
ansible is
a known, powerfull, tool that has been adopted by many operators
already. It'll
provide the flexibility needed and, if 

Re: [openstack-dev] [TripleO] Let's use Ansible to deploy OpenStack services on Kubernetes

[Jiří Stránský]

On 14.7.2017 11:17, Flavio Percoco wrote:


Greetings,

As some of you know, I've been working on the second phase of TripleO's
containerization effort. This phase if about migrating the docker based
deployment onto Kubernetes.

These phase requires work on several areas: Kubernetes deployment, OpenStack
deployment on Kubernetes, configuration management, etc. While I've been diving
into all of these areas, this email is about the second point, OpenStack
deployment on Kubernetes.

There are several tools we could use for this task. kolla-kubernetes,
openstack-helm, ansible roles, among others. I've looked into these tools and
I've come to the conclusion that TripleO would be better of by having ansible
roles that would allow for deploying OpenStack services on Kubernetes.

The existing solutions in the OpenStack community require using Helm. While I
like Helm and both, kolla-kubernetes and openstack-helm OpenStack projects, I
believe using any of them would add an extra layer of complexity to TripleO,
which is something the team has been fighting for years years - especially now
that the snowball is being chopped off.

Adopting any of the existing projects in the OpenStack communty would require
TripleO to also write the logic to manage those projects. For example, in the
case of openstack-helm, the TripleO team would have to write either ansible
roles or heat templates to manage - install, remove, upgrade - the charts (I'm
happy to discuss this point further but I'm keepping it at a high-level on
purpose for the sake of not writing a 10k-words-long email).

James Slagle sent an email[0], a couple of days ago, to form TripleO plans
around ansible. One take-away from this thread is that TripleO is adopting
ansible more and more, which is great and it fits perfectly with the conclusion
I reached.

Now, what this work means is that we would have to write an ansible role for
each service that will deploy the service on a Kubernetes cluster. Ideally these
roles will also generate the configuration files (removing the need of puppet
entirely) and they would manage the lifecycle. The roles would be isolated and
this will reduce the need of TripleO Heat templates. Doing this would give
TripleO full control on the deployment process too.

In addition, we could also write Ansible Playbook Bundles to contain these roles
and run them using the existing docker-cmd implementation that is coming out in
Pike (you can find a PoC/example of this in this repo[1]).

Now, I do realize the amount of work this implies and that this is my
opinion/conclusion. I'm sending this email out to kick-off the discussion and
gather thoughts and opinions from the rest of the community.


I agree this is a direction we should explore further. This would give 
us the option to tailor things exactly as we need -- good for keeping 
our balance in having interfaces as stable as possible, while still 
making enough development progress. And we'd keep our ability to make 
important changes (e.g. bugfixes) without delays.


We'll have to write more code ourselves, but it's possible that if we 
picked up an existing tool, we'd have to spend that time (if not more) 
elsewhere. Migrating existing non-kubernetized TripleO deployments to 
kubernetized is going to be pretty difficult even if we do what you 
suggested. I imagine that if we also had to fit into some pre-existing 
external deployment/management interfaces, while trying to keep ours 
stable or make just iterative changes, it might turn out to be a surreal 
effort. We will have to design things with migration from "legacy 
TripleO" in mind, or make later amendments here and there solely for 
this purpose. Such design and patches would probably not be a good fit 
for non-tripleo projects.


What i recall from our old PoC [2], defining the resources and init 
containers etc. will probably not be the most difficult task, and 
furthermore we can largely draw inspiration from our current 
containerized solution too. I think the more challenging things might be 
e.g. config generation with Ansible, and how major upgrades and rolling 
updates will be done (how all this ties into the APB way of 
provisioning/deprovisioning). And of course how to fulfill the 
expectations that TripleO has set around network isolation and HA :)


I'm eager to give the latest code a try myself :) Thanks for working on 
this, it looks like there's been great progress lately!


Jirka



Finally, what I really like about writing pure ansible roles is that ansible is
a known, powerfull, tool that has been adopted by many operators already. It'll
provide the flexibility needed and, if structured correctly, it'll allow for
operators (and other teams) to just use the parts they need/want without
depending on the full-stack. I like the idea of being able to separate concerns
in the deployment workflow and the idea of making it simple for users of TripleO
to do the same at runtime. Unfortunately, going down this road means 

[openstack-dev] [tripleo] containers-multinode-upgrades-nv is stable, please respect the results

[Jiří Stránský]

Hi all,

i'm just sending this plea -- let's pay attention to the 
containers-multinode-upgrades-nv job results in the CI please, and treat 
it as voting if possible. There's been a fair amount of breakage lately 
but all was caused by merging TripleO patches on which the job failed. 
The job in itself has been reliable.


We're on the way to make it voting, but this depends also on adding it 
to promotion jobs, so that RDO RPMs cannot be promoted if this job is 
failing (e.g. due to changes in non-TripleO projects which don't run the 
job in gerrit).



Thanks, and have a good day!

Jirka

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo] proposing Alex Schultz tripleo-core in all projects

[Jiří Stránský]

On 7.7.2017 19:39, Emilien Macchi wrote:

Alex has demonstrated high technical and community skills in TripleO -
where he's already core on THT, instack-undercloud, and puppet-tripleo
- but also very involved in other repos.
I propose that we extend his core status to all TripleO projects and
of course trust him (like we trust all core members) to review patches
were we feel confortable with.

He has shown an high interest in reviewed other TripleO projects and I
think he would be ready for this change.
As usual, this is an open proposal, any feedback is welcome.

Thanks,



+1

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo] CI Squad Meeting Summary (week 26) - job renaming discussion

[Jiří Stránský]

On 30.6.2017 17:06, Jiří Stránský wrote:

On 30.6.2017 15:04, Attila Darazs wrote:

= Renaming the CI jobs =

When we started the job transition to Quickstart, we introduced the
concept of featuresets[1] that define a certain combination of features
for each job.

This seemed to be a sensible solution, as it's not practical to mention
all the individual features in the job name, and short names can be
misleading (for example ovb-ha job does so much more than tests HA).

We decided to keep the original names for these jobs to simplify the
transition, but the plan is to rename them to something that will help
to reproduce the jobs locally with Quickstart.

The proposed naming scheme will be the same as the one we're now using
for job type in project-config:

gate-tripleo-ci-centos-7-{node-config}-{featureset-config}

So for example the current "gate-tripleo-ci-centos-7-ovb-ha-oooq" job
would look like "gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001"


I'd prefer to keep the job names somewhat descriptive... If i had to
pick one or the other, i'd rather stick with the current way, as at
least for me it's higher priority to see descriptive names in CI results
than saving time on finding featureset file mapping when needing to
reproduce a job result. My eyes scan probably more than a hundred of
individual CI job results daily, but i only need to reproduce 0 or 1 job
failures locally usually.

Alternatively, could we rename "featureset001.yaml" into
"featureset-ovb-ha.yaml" and then have i guess something like
"gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-ovb-ha" for the job name?
Maybe "ovb" would be there twice, in case it's needed both in node
config and featureset parts of the job name...

Or we could pull the mapping between job name and job type in an
automated way from project-config.


^ I mean for the purposes of reproducing a CI job, in a similar way we 
do it for running the CI job in the first place.




(Will be on PTO for a week from now, apologies if i don't respond timely
here.)


Have a good day,

Jirka



The advantage of this will be that it will be easy to reproduce a gate
job on a local virthost by typing something like:

./quickstart.sh --release tripleo-ci/master \
   --nodes config/nodes/3ctlr_1comp.yml \
   --config config/general_config/featureset001.yml \
   

Please let us know if this method sounds like a step forward.



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo] CI Squad Meeting Summary (week 26) - job renaming discussion

[Jiří Stránský]

On 30.6.2017 15:04, Attila Darazs wrote:

= Renaming the CI jobs =

When we started the job transition to Quickstart, we introduced the
concept of featuresets[1] that define a certain combination of features
for each job.

This seemed to be a sensible solution, as it's not practical to mention
all the individual features in the job name, and short names can be
misleading (for example ovb-ha job does so much more than tests HA).

We decided to keep the original names for these jobs to simplify the
transition, but the plan is to rename them to something that will help
to reproduce the jobs locally with Quickstart.

The proposed naming scheme will be the same as the one we're now using
for job type in project-config:

gate-tripleo-ci-centos-7-{node-config}-{featureset-config}

So for example the current "gate-tripleo-ci-centos-7-ovb-ha-oooq" job
would look like "gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001"


I'd prefer to keep the job names somewhat descriptive... If i had to 
pick one or the other, i'd rather stick with the current way, as at 
least for me it's higher priority to see descriptive names in CI results 
than saving time on finding featureset file mapping when needing to 
reproduce a job result. My eyes scan probably more than a hundred of 
individual CI job results daily, but i only need to reproduce 0 or 1 job 
failures locally usually.


Alternatively, could we rename "featureset001.yaml" into 
"featureset-ovb-ha.yaml" and then have i guess something like 
"gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-ovb-ha" for the job name? 
Maybe "ovb" would be there twice, in case it's needed both in node 
config and featureset parts of the job name...


Or we could pull the mapping between job name and job type in an 
automated way from project-config.


(Will be on PTO for a week from now, apologies if i don't respond timely 
here.)



Have a good day,

Jirka



The advantage of this will be that it will be easy to reproduce a gate
job on a local virthost by typing something like:

./quickstart.sh --release tripleo-ci/master \
  --nodes config/nodes/3ctlr_1comp.yml \
  --config config/general_config/featureset001.yml \
  

Please let us know if this method sounds like a step forward.


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [deployment][kolla][openstack-ansible][openstack-helm][tripleo] ansible role to produce oslo.config files for openstack services

[Jiří Stránský]

On 15.6.2017 19:06, Emilien Macchi wrote:

I missed [tripleo] tag.

On Thu, Jun 15, 2017 at 12:09 PM, Emilien Macchi  wrote:

If you haven't followed the "Configuration management with etcd /
confd" thread [1], Doug found out that using confd to generate
configuration files wouldn't work for the Cinder case where we don't
know in advance of the deployment what settings to tell confd to look
at.
We are still looking for a generic way to generate *.conf files for
OpenStack, that would be usable by Deployment tools and operators.
Right now, Doug and I are investigating some tooling that would be
useful to achieve this goal.

Doug has prototyped an Ansible role that would generate configuration
files by consumming 2 things:

* Configuration schema, generated by Ben's work with Machine Readable
Sample Config.
   $ oslo-config-generator --namespace cinder --format yaml > cinder-schema.yaml

It also needs: https://review.openstack.org/#/c/474306/ to generate
some extra data not included in the original version.

* Parameters values provided in config_data directly in the playbook:
config_data:
  DEFAULT:
transport_url: rabbit://user:password@hostname
verbose: true

There are 2 options disabled by default but which would be useful for
production environments:
* Set to true to always show all configuration values: config_show_defaults
* Set to true to show the help text: config_show_help: true

The Ansible module is available on github:
https://github.com/dhellmann/oslo-config-ansible

To try this out, just run:
   $ ansible-playbook ./playbook.yml

You can quickly see the output of cinder.conf:
 https://clbin.com/HmS58


What are the next steps:

* Getting feedback from Deployment Tools and operators on the concept
of this module.
   Maybe this module could replace what is done by Kolla with
merge_configs and OpenStack Ansible with config_template.
* On the TripleO side, we would like to see if this module could
replace the Puppet OpenStack modules that are now mostly used for
generating configuration files for containers.
   A transition path would be having Heat to generate Ansible vars
files and give it to this module. We could integrate the playbook into
a new task in the composable services, something like
   "os_gen_config_tasks", a bit like we already have for upgrade tasks,
also driven by Ansible.


This sounds good to me, though one issue i can presently see is that 
Puppet modules sometimes contain quite a bit of data processing logic 
("smart" variables which map 1-to-N rather than 1-to-1 to actual config 
values, and often not just in openstack service configs, e.g. 
puppet-nova also configures libvirt, etc.). Also we use some non-config 
aspects from the Puppet modules (e.g. seeding Keystone 
tenants/services/endpoints/...). We'd need to implement this 
functionality elsewhere when replacing the Puppet modules. Not a 
blocker, but something to keep in mind.



* Another similar option to what Doug did is to write a standalone
tool that would generate configuration, and for Ansible users we would
write a new module to use this tool.
   Example:
   Step 1. oslo-config-generator --namespace cinder --format yaml >
cinder-schema.yaml (note this tool already exists)
   Step 2. Create config_data.yaml in a specific format with
parameters values for what we want to configure (note this format
doesn't exist yet but look at what Doug did in the role, we could use
the same kind of schema).
   Step 3. oslo-gen-config -i config_data.yaml -s schema.yaml >
cinder.conf (note this tool doesn't exist yet)


+1 on standalone tool which can be used in different contexts (by 
different higher level tools), this sounds generally useful.




   For Ansible users, we would write an Ansible module that would
take in entry 2 files: the schema and the data. The module would just
run the tool provided by oslo.config.
   Example:
   - name: Generate cinder.conf
 oslo-gen-config: schema=cinder-schema.yaml
data=config_data.yaml


+1 for module rather than a role. "Take these inputs and produce that 
output" fits the module semantics better than role semantics IMO.


FWIW as i see it right now, this ^^ + ConfigMaps + immutable-config 
containers could result in a nicer/safer/more-debuggable containerized 
OpenStack setup than etcd + confd in daemon mode + mutable-config 
containers.





Please bring feedback and thoughts, it's really important to know what
folks from Installers think about this idea; again the ultimate goal
is to provide a reference tool to generate configuration in OpenStack,
in a way that scales and is friendly for our operators.

Thanks,

[1] http://lists.openstack.org/pipermail/openstack-dev/2017-June/118176.html
--
Emilien Macchi






Have a good day,

Jirka

__
OpenStack Development Mailing List (not for usage 

Re: [openstack-dev] [tripleo] Containers Deep Dive - 15th June

[Jiří Stránský]

On 15.6.2017 18:25, Jiří Stránský wrote:

On 9.6.2017 16:49, Jiří Stránský wrote:

Hello,

as discussed previously on the list and at the weekly meeting, we'll do
a deep dive about containers. The time:

Thursday 15th June, 14:00 UTC (the usual time)

Link for attending will be at the deep dives etherpad [1], preliminary
agenda is in another etherpad [2], and i hope i'll be able to record it too.


The recording is available here:

https://www.youtube.com/watch?v=xhTwHfi65p8


And the slides:

https://jistr.github.io/tripleo-deep-dive-containers-june-2017/



Thanks Carlos for managing our youtube channel!

Jirka



This time it may be more of a "broad dive" :) as that's what containers
in TripleO mostly are -- they add new bits into many TripleO
areas/topics (composable services/upgrades, Quickstart/CI, etc.). So
i'll be trying to bring light to the container-specific parts of the
mix, and assume some familiarity with the generic TripleO
concepts/features (e.g. via docs and previous deep dives). Given this
pattern, i'll have slides with links into code. I'll post them online,
so that you can reiterate or examine some code more closely later, in
case you want to.


Have a good day!

Jirka

[1] https://etherpad.openstack.org/p/tripleo-deep-dive-topics
[2] https://etherpad.openstack.org/p/tripleo-deep-dive-containers

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo] Containers Deep Dive - 15th June

[Jiří Stránský]

On 9.6.2017 16:49, Jiří Stránský wrote:

Hello,

as discussed previously on the list and at the weekly meeting, we'll do
a deep dive about containers. The time:

Thursday 15th June, 14:00 UTC (the usual time)

Link for attending will be at the deep dives etherpad [1], preliminary
agenda is in another etherpad [2], and i hope i'll be able to record it too.


The recording is available here:

https://www.youtube.com/watch?v=xhTwHfi65p8

Thanks Carlos for managing our youtube channel!

Jirka



This time it may be more of a "broad dive" :) as that's what containers
in TripleO mostly are -- they add new bits into many TripleO
areas/topics (composable services/upgrades, Quickstart/CI, etc.). So
i'll be trying to bring light to the container-specific parts of the
mix, and assume some familiarity with the generic TripleO
concepts/features (e.g. via docs and previous deep dives). Given this
pattern, i'll have slides with links into code. I'll post them online,
so that you can reiterate or examine some code more closely later, in
case you want to.


Have a good day!

Jirka

[1] https://etherpad.openstack.org/p/tripleo-deep-dive-topics
[2] https://etherpad.openstack.org/p/tripleo-deep-dive-containers

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo] Containers Deep Dive - 15th June

[Jiří Stránský]

On 9.6.2017 16:49, Jiří Stránský wrote:

Hello,

as discussed previously on the list and at the weekly meeting, we'll do
a deep dive about containers. The time:

Thursday 15th June, 14:00 UTC (the usual time)

Link for attending will be at the deep dives etherpad [1], preliminary
agenda is in another etherpad [2], and i hope i'll be able to record it too.

This time it may be more of a "broad dive" :) as that's what containers
in TripleO mostly are -- they add new bits into many TripleO
areas/topics (composable services/upgrades, Quickstart/CI, etc.). So
i'll be trying to bring light to the container-specific parts of the
mix, and assume some familiarity with the generic TripleO
concepts/features (e.g. via docs and previous deep dives). Given this
pattern, i'll have slides with links into code. I'll post them online,
so that you can reiterate or examine some code more closely later, in
case you want to.


For folks who haven't had any prior exposure to Docker containers 
whatsoever, i'd recommend giving these links a scan beforehand:


* Docker Overview
  https://docs.docker.com/engine/docker-overview/

* Docker - Get Started pt.1: Orientation and Setup
  https://docs.docker.com/get-started/

* Docker - Get Started pt.2: Containers
  https://docs.docker.com/get-started/part2/

(I'd like us to spend majority of the time talking about how we use 
containers in TripleO, rather than what containers are.)



Thanks, looking forward to seeing you at the deep dive!

Jirka




Have a good day!

Jirka

[1] https://etherpad.openstack.org/p/tripleo-deep-dive-topics
[2] https://etherpad.openstack.org/p/tripleo-deep-dive-containers

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo] Containers Deep Dive - 15th June

[Jiří Stránský]

On 13.6.2017 09:58, Or Idgar wrote:

Hi,
Can you please send me the meeting invitation?


Hi Or,

i've just added the bluejeans link to the etherpads. I'm not posting it 
here as i want the "sources of truth" for that link to be editable, in 
case we hit some issues with the setup / joining etc.


Jirka




Thanks in advance!

On Fri, Jun 9, 2017 at 5:49 PM, Jiří Stránský <ji...@redhat.com> wrote:


Hello,

as discussed previously on the list and at the weekly meeting, we'll do a
deep dive about containers. The time:

Thursday 15th June, 14:00 UTC (the usual time)

Link for attending will be at the deep dives etherpad [1], preliminary
agenda is in another etherpad [2], and i hope i'll be able to record it too.

This time it may be more of a "broad dive" :) as that's what containers in
TripleO mostly are -- they add new bits into many TripleO areas/topics
(composable services/upgrades, Quickstart/CI, etc.). So i'll be trying to
bring light to the container-specific parts of the mix, and assume some
familiarity with the generic TripleO concepts/features (e.g. via docs and
previous deep dives). Given this pattern, i'll have slides with links into
code. I'll post them online, so that you can reiterate or examine some code
more closely later, in case you want to.


Have a good day!

Jirka

[1] https://etherpad.openstack.org/p/tripleo-deep-dive-topics
[2] https://etherpad.openstack.org/p/tripleo-deep-dive-containers

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev







__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [deployment] [oslo] [ansible] [tripleo] [kolla] [helm] Configuration management with etcd / confd

[Jiří Stránský]

On 9.6.2017 18:51, Flavio Percoco wrote:

A-ha, ok! I figured this was another option. In this case I guess we would
have 2 options:

1. Run confd + openstack service in side the container. My concern in this
case
would be that we'd have to run 2 services inside the container and structure
things in a way we can monitor both services and make sure they are both
running. Nothing impossible but one more thing to do.


I see several cons with this option:

* Even if we do this in a sidecar container like Bogdan mentioned (which 
is better than running 2 "top-level" processes in a single container 
IMO), we still have to figure out when to restart the main service, 
IIUC. I see confd in daemon mode listens on the backend change and 
updates the conf files, but i can't find a mention that it would be able 
to restart services. Even if we implemented this auto-restarting in 
OpenStack services, we need to deal with services like MariaDB, Redis, 
..., so additional wrappers might be needed to make this a generic solution.


* Assuming we've solved the above, if we push a config change to etcd, 
all services get restarted at roughly the same time, possibly creating 
downtime or capacity issues.


* It complicates the reasoning about container lifecycle, as we have to 
start distinguishing between changes that don't require a new container 
(config change only) vs. changes which do require it (image content 
change). Mutable container config also hides this lifecycle from the 
operator -- the container changes on the inside without COE knowing 
about it, so any operator's queries to COE would look like no changes 
happened.


I think ideally container config would be immutable, and every time we 
want to change anything, we'd do that via a roll out of a new set of 
containers. This way we have a single way of making changes to reason 
about, and when we're doing rolling updates, it shouldn't result in a 
downtime or tangible performance drop. (Not talking about migrating to a 
new major OpenStack release, which will still remain a special case in 
foreseeable future.)




2. Run confd `-onetime` and then run the openstack service.


This sounds simpler both in terms of reasoning and technical complexity, 
so if we go with confd, i'd lean towards this option. We'd have to 
rolling-replace the containers from outside, but that's what k8s can 
take care of, and at least the operator can see what's happening on high 
level.


The issues that Michał mentioned earlier still remain to be solved -- 
config versioning ("accidentally" picking up latest config), and how to 
supply config elements that differ per host.


Also, it's probably worth diving a bit deeper into comparing `confd 
-onetime` and ConfigMaps...



Jirka




Either would work but #2 means we won't have config files monitored and the
container would have to be restarted to update the config files.

Thanks, Doug.
Flavio



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [tripleo] Containers Deep Dive - 15th June

[Jiří Stránský]

Hello,

as discussed previously on the list and at the weekly meeting, we'll do 
a deep dive about containers. The time:


Thursday 15th June, 14:00 UTC (the usual time)

Link for attending will be at the deep dives etherpad [1], preliminary 
agenda is in another etherpad [2], and i hope i'll be able to record it too.


This time it may be more of a "broad dive" :) as that's what containers 
in TripleO mostly are -- they add new bits into many TripleO 
areas/topics (composable services/upgrades, Quickstart/CI, etc.). So 
i'll be trying to bring light to the container-specific parts of the 
mix, and assume some familiarity with the generic TripleO 
concepts/features (e.g. via docs and previous deep dives). Given this 
pattern, i'll have slides with links into code. I'll post them online, 
so that you can reiterate or examine some code more closely later, in 
case you want to.



Have a good day!

Jirka

[1] https://etherpad.openstack.org/p/tripleo-deep-dive-topics
[2] https://etherpad.openstack.org/p/tripleo-deep-dive-containers

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO] overcloud containers patches todo

[Jiří Stránský]

On 5.6.2017 23:52, Dan Prince wrote:

On Mon, 2017-06-05 at 16:11 +0200, Jiří Stránský wrote:

On 5.6.2017 08:59, Sagi Shnaidman wrote:

Hi
I think a "deep dive" about containers in TripleO and some helpful
documentation would help a lot for valuable reviews of these
container
patches. The knowledge gap that's accumulated here is pretty big.


As per last week's discussion [1], i hope this is something i could
do.
I'm drafting a preliminary agenda in this etherpad, feel free to add
more suggestions if i missed something:

https://etherpad.openstack.org/p/tripleo-deep-dive-containers

My current intention is to give a fairly high level view of the
TripleO
container land: from deployment, upgrades, debugging failed CI jobs,
to
how CI itself was done.

I'm hoping we could make it this Thursday still. If that's too short
of
a notice for several folks or if i hit some trouble with preparation,
we
might move it to 15th. Any feedback is welcome of course.


Nice Jirka. Thanks for organizing this!

Dan


Sure thing. I'll do it on 15th indeed as a couple more topics appeared 
and i'd like to familiarize myself with some details (alongside doing 
normal work :) ).


Jirka





Have a good day,

Jirka



Thanks

On Jun 5, 2017 03:39, "Dan Prince" <dpri...@redhat.com> wrote:


Hi,

Any help reviewing the following patches for the overcloud
containerization effort in TripleO would be appreciated:

https://etherpad.openstack.org/p/tripleo-containers-todo

If you've got new services related to the containerization
efforts feel
free to add them here too.

Thanks,

Dan

_
_
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:un
subscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev





___
___
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsu
bscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




_
_
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubs
cribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO] overcloud containers patches todo

[Jiří Stránský]

On 5.6.2017 08:59, Sagi Shnaidman wrote:

Hi
I think a "deep dive" about containers in TripleO and some helpful
documentation would help a lot for valuable reviews of these container
patches. The knowledge gap that's accumulated here is pretty big.


As per last week's discussion [1], i hope this is something i could do. 
I'm drafting a preliminary agenda in this etherpad, feel free to add 
more suggestions if i missed something:


https://etherpad.openstack.org/p/tripleo-deep-dive-containers

My current intention is to give a fairly high level view of the TripleO 
container land: from deployment, upgrades, debugging failed CI jobs, to 
how CI itself was done.


I'm hoping we could make it this Thursday still. If that's too short of 
a notice for several folks or if i hit some trouble with preparation, we 
might move it to 15th. Any feedback is welcome of course.


Have a good day,

Jirka



Thanks

On Jun 5, 2017 03:39, "Dan Prince"  wrote:


Hi,

Any help reviewing the following patches for the overcloud
containerization effort in TripleO would be appreciated:

https://etherpad.openstack.org/p/tripleo-containers-todo

If you've got new services related to the containerization efforts feel
free to add them here too.

Thanks,

Dan

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev





__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [EXTERNAL] Re: [TripleO] custom configuration to overcloud fails second time

[Jiří Stránský]

On 31.5.2017 17:40, Dnyaneshwar Pawar wrote:

Hi Ben,

On 5/31/17, 8:06 PM, "Ben Nemec" 
> wrote:

I think we would need to see what your custom config templates look like
as well.

Custom config templates: http://paste.openstack.org/show/64/


Hello Dnyaneshwar,

from a brief scan of that paste i think that:

  OS::TripleO::ControllerExtraConfig: /home/stack/example_2.yaml

should rather be:

  OS::TripleO::ControllerExtraConfigPre: /home/stack/example_2.yaml


The 'Pre' hook gets a `server` parameter (not `servers`) - it's 
instantiated per server [1], not per role. There are docs [2] that 
describe the interface, and they describe some alternative options as well.


(Please ask such questions on IRC channel #tripleo on freenode, as the 
openstack-dev list is meant mainly for development discussion.)



Have a good day,

Jirka

[1] 
https://github.com/openstack/tripleo-heat-templates/blob/b344f5994fcd16e562d55e6e00ad0980c5b32621/puppet/role.role.j2.yaml#L475-L479

[2] http://tripleo.org/advanced_deployment/extra_config.html




Also note that it's generally not recommended to drop environment files
from your deploy command unless you explicitly want to stop applying
them.  So if you applied myconfig_1.yaml and then later want to apply
myconfig_2.yaml your deploy command should look like: openstack
overcloud deploy --templates -e myconfig_1.yaml -e myconfig_2.yaml

Yes, I agree. But in my case even if I dropped myconfig_1.yaml while applying 
myconfig_2.yaml , config in step 1 remained unchanged.

On 05/31/2017 07:53 AM, Dnyaneshwar Pawar wrote:
Hi TripleO Experts,
I performed following steps -

   1. openstack overcloud deploy --templates -e myconfig_1.yaml
   2. openstack overcloud deploy --templates -e myconfig_2.yaml

Step 1  Successfully applied custom configuration to the overcloud.
Step 2 completed successfully but custom configuration is not applied to
the overcloud. And configuration applied by step 1 remains unchanged.

*Do I need to do anything before performing step 2?*


Thanks and Regards,
Dnyaneshwar


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: 
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: 
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo] CI Squad Meeting Summary (week 21) - Devmode OVB, RDO Cloud and config management

[Jiří Stránský]

On 30.5.2017 23:03, Emilien Macchi wrote:

On Fri, May 26, 2017 at 4:58 PM, Attila Darazs  wrote:

If the topics below interest you and you want to contribute to the
discussion, feel free to join the next meeting:

Time: Thursdays, 14:30-15:30 UTC
Place: https://bluejeans.com/4113567798/

Full minutes: https://etherpad.openstack.org/p/tripleo-ci-squad-meeting

= Periodic & Promotion OVB jobs Quickstart transition =

We had a lively technical discussions this week. Gabriele's work on
transitioning the periodic & promotion jobs is nearly complete, only needs
reviews at this point. We won't set a transition date for these as it is not
really impacting folks long term if these jobs are failing for a few days at
this point. We'll transition when everything is ready.

= RDO Cloud & Devmode OVB =

We continued planning the introduction of RDO Cloud for the upstream OVB
jobs. We're still at the point of account setup.

The new OVB based devmode seems to be working fine. If you have access to
RDO Cloud, and haven't tried it already, give it a go. It can set up a full
master branch based deployment within 2 hours, including any pending changes
baked into the under & overcloud.

When you have your account info sourced, all it takes is

$ ./devmode.sh --ovb

from your tripleo-quickstart repo! See here[1] for more info.

= Container jobs on nodepool multinode =

Gabriele is stuck with these new Quickstart jobs. We would need a deep dive
into debugging and using the container based TripleO deployments. Let us
know if you can do one!


I've pinged some folks around, let's see if someone volunteers to make it.


I can lead the deep dive as i've been involved in implementation of all 
the multinode jobs (deployment, upgrade, scenarios deployment + upgrade).


Currently only the upgrade job is merged and working. Deployment job is 
ready to merge (imho :) ), let's get it in place before we do the deep 
dive, so that we talk about things that are already working as intended. 
However, i think we don't have to block on getting the scenario jobs all 
going and polished, those are not all green yet (2/4 deployment jobs 
green, 1/4 upgrade jobs green).


I hope that sounds like a sensible plan. Let me know in case you have 
any feedback.


Thanks

Jirka




= How to handle Quickstart configuration =

This a never-ending topic, on which we managed to spend a good chunk of time
this week as well. Where should we put various configs? Should we duplicate
a bunch of variables or cut them into small files?

For now it seems we can agree on 3 levels of configuration:

* nodes config (i.e. how many nodes we want for the deployment)
* envionment + provisioner settings (i.e. you want to run on rdocloud with
ovb, or on a local machine with libvirt)
* featureset (a certain set of features enabled/disabled for the jobs, like
pacemaker and ssl)

This seems rather straightforward until we encounter exceptions. We're going
to figure out the edge cases and rework the current configs to stick to the
rules.


That's it for this week. Thank you for reading the summary.

Best regards,
Attila

[1] http://docs.openstack.org/developer/tripleo-quickstart/devmode-ovb.html

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev







__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO] Undercloud backup and restore

[Jiří Stránský]

On 24.5.2017 15:02, Marios Andreou wrote:

On Wed, May 24, 2017 at 10:26 AM, Carlos Camacho Gonzalez <
ccama...@redhat.com> wrote:


Hey folks,

Based on what we discussed yesterday in the TripleO weekly team meeting,
I'll like to propose a blueprint to create 2 features, basically to backup
and restore the Undercloud.

I'll like to follow in the first iteration the available docs for this
purpose [1][2].

With the addition of backing up the config files on /etc/ specifically to
be able to recover from a failed Undercloud upgrade, i.e. recover the repos
info removed in [3].

I'll like to target this for P as I think I have enough time for
coding/testing these features.

I already have created a blueprint to track this effort
https://blueprints.launchpad.net/tripleo/+spec/undercloud-backup-restore

What do you think about it?



+1 from me as you know but adding my support on the list too. I think it is
a great idea - there are cases especially around changing network config
during an upgrade for example where the best remedy is to restore the
undercloud for the network definitions (both neutron config and heat db).


+1 i think there's not really an easy way out of these issues other than 
a restore. We already recommend doing a backup before upgrading [1], so 
having something that can further help operators in this regard would be 
good.


Jirka

[1] http://tripleo.org/post_deployment/upgrade.html



thanks,




Thanks,
Carlos.

[1]: https://access.redhat.com/documentation/en-us/red_hat_
enterprise_linux_openstack_platform/7/html/back_up_and_
restore_red_hat_enterprise_linux_openstack_platform/restore

[2]: https://docs.openstack.org/developer/tripleo-docs/post_
deployment/backup_restore_undercloud.html

[3]: https://docs.openstack.org/developer/tripleo-docs/
installation/updating.html


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev






__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [tripleo][ci] Saving memory in OOOQ multinode jobs

[Jiří Stránský]

Hi all,

we can save some memory in our OOOQ multinode jobs by specifying custom 
role data -- getting rid of resources and YAQL crunching for roles that 
aren't used at all in the end. Shout out to shardy for suggesting we 
should look into doing this.


First observations, hopefully at least somewhat telling:

nonha-multinode-oooq job: comparing [1][2] the RSS memory usage by the 4 
heat-engine processes on undercloud drops from 803 MB to 690 MB.


containers-multinode-upgrades job: comparing [3][4] heat-engine memory 
usage drops from 1221 MB to 968 MB.


I expected some time savings as well but wasn't able to spot any, looks 
like concurrency works well in Heat :)


Patches are here:
https://review.openstack.org/#/c/455730/
https://review.openstack.org/#/c/455719/


Have a good day,

Jirka


[1] 
http://logs.openstack.org/68/465468/1/check/gate-tripleo-ci-centos-7-nonha-multinode-oooq/9d354b5/logs/undercloud/var/log/host_info.txt.gz
[2] 
http://logs.openstack.org/30/455730/3/check/gate-tripleo-ci-centos-7-nonha-multinode-oooq/4e3bb4a/logs/undercloud/var/log/host_info.txt.gz
[3] 
http://logs.openstack.org/52/464652/5/experimental/gate-tripleo-ci-centos-7-containers-multinode-upgrades-nv/a20f7dd/logs/undercloud/var/log/host_info.txt.gz
[4] 
http://logs.openstack.org/30/455730/3/experimental/gate-tripleo-ci-centos-7-containers-multinode-upgrades-nv/f753cd9/logs/undercloud/var/log/host_info.txt.gz


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo][ci] Upgrade CI job for O->P (containerization)

[Jiří Stránský]

On 12.5.2017 15:30, Emilien Macchi wrote:

On Wed, May 10, 2017 at 9:26 AM, Jiří Stránský <ji...@redhat.com> wrote:

Hi all,

the upgrade job which tests Ocata -> Pike/master upgrade (from bare-metal to
containers) just got a green flag from the CI [1].

I've listed the remaining patches we need to land at the very top of the
container CI etherpad [2], please let's get them reviewed and landed as soon
as we can. The sooner we get the job going, the fewer upgrade regressions
will get merged in the meantime (e.g. we have one from last week).

The CI job utilizes mixed release deployment (master undercloud, overcloud
deployed as Ocata and upgraded to latest). It tests the main overcloud
upgrade phase (no separate compute role upgrades, no converge phase). This
means the testing isn't exhaustive to the full expected "production
scenario", but it covers the most important part where we're likely to see
the most churn and potential breakages. We'll see how much spare wall time
we have to add more things once we get the job to run on patches regularly.


The job you and the team made to make that happen is amazing and outstanding.


Thanks! I should mention i've utilized quite a lot of pre-existing 
upgrades skeleton code in OOOQ that i think mainly Mathieu Bultel put 
together.



Once the jobs are considered stable, I would move them to the gate so
we don't break it. Wdyt?


Yes absolutely, i'd start with non-voting as usual first.

We still need reviews on the patches to move forward (4 patches 
remaining), so when you (plural :) ) have some bandwidth, please take a 
look:


https://review.openstack.org/#/c/462664/ - 
multinode-container-upgrade.yaml usable for mixed upgrade


https://review.openstack.org/#/c/459789/ - TripleO CI mixed release 
master UC / ocata OC


https://review.openstack.org/#/c/462172/  - Don't overwrite tht_dir for 
upgrades to master


https://review.openstack.org/#/c/460061/ - Use mixed release for 
container upgrade



Thanks

Jirka






Thanks and have a good day!

Jirka

[1]
http://logs.openstack.org/61/460061/15/experimental/gate-tripleo-ci-centos-7-containers-multinode-upgrades-nv/d7faa50/
[2] https://etherpad.openstack.org/p/tripleo-containers-ci

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev







__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [tripleo][ci] Upgrade CI job for O->P (containerization)

[Jiří Stránský]

Hi all,

the upgrade job which tests Ocata -> Pike/master upgrade (from 
bare-metal to containers) just got a green flag from the CI [1].


I've listed the remaining patches we need to land at the very top of the 
container CI etherpad [2], please let's get them reviewed and landed as 
soon as we can. The sooner we get the job going, the fewer upgrade 
regressions will get merged in the meantime (e.g. we have one from last 
week).


The CI job utilizes mixed release deployment (master undercloud, 
overcloud deployed as Ocata and upgraded to latest). It tests the main 
overcloud upgrade phase (no separate compute role upgrades, no converge 
phase). This means the testing isn't exhaustive to the full expected 
"production scenario", but it covers the most important part where we're 
likely to see the most churn and potential breakages. We'll see how much 
spare wall time we have to add more things once we get the job to run on 
patches regularly.



Thanks and have a good day!

Jirka

[1] 
http://logs.openstack.org/61/460061/15/experimental/gate-tripleo-ci-centos-7-containers-multinode-upgrades-nv/d7faa50/

[2] https://etherpad.openstack.org/p/tripleo-containers-ci

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo][kolla] extended start and hostpath persistent dirs

[Jiří Stránský]

Responses inline :) I started snipping parts out because the quotations 
are getting long.





tl;dr use kolla images and bootsrap OR upstream images with
direct
commands:

.. code-block:: yaml
kolla_config:
/var/lib/kolla/config_files/foo.json
  command: /usr/bin/foo


We insist on using kolla extended start, if only it can't function
w/o Kolla extended start using Kolla images (like Mysql and Rabbitmq
have some extra initialization)



I don't think we need to insist on this, in fact i'd even prefer not 
using kolla_start in cases where it doesn't provide significant value.


In addition to the Kolla flavor that Dan mentioned earlier, using these 
entrypoints also increases chance of TripleO being broken by a commit to 
Kolla, because Kolla commits aren't gated on deploying TripleO.


If the only thing we need is the file permissions setup, we don't even 
need to use kolla_start for that, we can use kolla_set_configs [4] 
directly in an init container. (But again, i don't think we need to 
force this as a rule.)






vs

.. code-block:: yaml
foo:
image: upstream/foo:latest
command: /usr/bin/foo


We accept direct commands as well, if it works w/o "user: root" for
Kolla containers omitting extended start OR if it just works as is with
upstream containers (non Kolla), like etcd [2] and perhaps redis


I think using direct commands is fine.

However, i think we should avoid targeting any images that we can't 
build ourselves easily. One of the benefits of using Kolla images is a 
uniform way how to build them.





* Custom entrypoints for containers adds complexity and head ache.


Good point. But the entry points Kolla uses for many containers don't
match what our systemd services already use on baremetal. As we are
striving for update path that does not break end users upgrading from
baremetal to containers we have to have a mechanism that gives us
configuration parity across the implementions. Controlling the entry
point either by injecting it into the container (via something like
Kolla's template overrides mechanism) or via tripleo-heat-templates
direction (much more hackable) is where we ended up.


Yea i'd very much prefer using entrypoints that are easily amendable by 
TripleO developers, and are gated on deploying TripleO.


As for having them in-image or externally passed from t-h-t, that could 
almost be a thread on its own :) The benefit of t-h-t approach is 
hackability. The benefit of in-image approach is being sure that the 
image version is compatible with its entrypoint (for rollbacks and such) 
and generally the image being more self contained (being able to use it 
easier manually, without Heat/t-h-t, should the need arise).


I think we may still be forming our opinion on this matter as we hit 
issues with one or the other approach, maybe we'll even continue using 
different approaches for different use cases.




In general we like Kolla images at the moment for what they provide.
But there are some cases where we need to control things that have too
much of a "kolla flavor" and would potentially break upgrades/features
if we used them directly.



We accept custom entry points for some tricky cases as well.

Having that said, I'd really like to see a final call for that topic.
Otherwise it's really hard to do a code review and perhaps to maintain
changes to t-h-t for future releases as well.




Cheers

Jirka

[4] 
https://github.com/openstack/kolla/blob/77903c70cd651ad97a6a918f6889e5120d85b8d1/docker/base/start.sh#L14


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo] Roadmap for Container CI work

[Jiří Stránský]

On 6.4.2017 12:46, Jiří Stránský wrote:

On 4.4.2017 22:01, Emilien Macchi wrote:

After our weekly meeting of today, I found useful to share and discuss
our roadmap for Container CI jobs in TripleO.
They are ordered by priority from the highest to lowest:

1. Swap ovb-nonha job with ovb-containers, enable introspection on the
container job and shuffle other coverage (e.g ssl) to other jobs
(HA?). It will help us to get coverage for ovb-containers scenario
again, without consuming more rh1 resources and keep existing
coverage.
2. Get multinode coverage of deployments - this should integrate with
the scenarios we already have defined for non-container deployment.
This is super important to cover all overcloud services, like we did
with classic deployments. It should be non voting to start and then
voting once it works. We should find a way to keep the same templates
as we have now, and just include the docker environment. In other
words, find a way to keep using:
https://github.com/openstack/tripleo-heat-templates/blob/master/ci/environments/scenario001-multinode.yaml
so we don't duplicate scenario environments.
3. Implement container upgrade job, which for Pike will be deploy a
baremetal overcloud, then migrate on upgrade to containers. Use
multinode jobs for this task. Start with a non-voting job and move to
the gate once it work. I also suggest to use scenarios framework, so
we keep good coverage.


The first iteration of this job is ready to be reviewed and landed.
Please see the patches here [1].

The latest job execution didn't go all the way to success yet, it failed
during Ansible upgrade steps execution [2], but i think the patches are
now far enough that they would be good to merge anyway, and issues can
be ironed out subsequently, as well as making the job actually
Ocata->master rather than master->master (currently just switching from
non-containers to containers).

[1] https://review.openstack.org/#/q/topic:container-upgrade
[2]
http://logs.openstack.org/84/450784/8/experimental/gate-tripleo-ci-centos-7-containers-multinode-upgrades-nv/a1850f7/logs/undercloud/home/jenkins/overcloud_deploy.log.txt.gz


Sorry the [2] link was incorrect, this is the right one:

http://logs.openstack.org/84/450784/9/experimental/gate-tripleo-ci-centos-7-containers-multinode-upgrades-nv/23f9190/logs/undercloud/home/jenkins/overcloud_upgrade_console.log.txt.gz




4. After we implement the workflow for minor updates, have a job with
tests container-to-container updates for minor (rolling) updates, this
ideally should add some coverage to ensure no downtime of APIs and
possibly checks for service restarts (ref recent bugs about bouncing
services on minor updates)
5. Once Pike is released and Queens starts, let's work on container to
containers upgrade job.

Any feedback or question is highly welcome,

Note: The proposal comes from shardy's notes on
https://etherpad.openstack.org/p/tripleo-container-ci - feel free to
contribute to the etherpad or mailing list.

Thanks,






__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo] Proposing Florian Fuchs for tripleo-validations core

[Jiří Stránský]

On 6.4.2017 11:53, Martin André wrote:

Hellooo,

I'd like to propose we extend Florian Fuchs +2 powers to the
tripleo-validations project. Florian is already core on tripleo-ui
(well, tripleo technically so this means there is no changes to make
to gerrit groups).

Florian took over many of the stalled patches in tripleo-validations
and is now the principal contributor in the project [1]. He has built
a good expertise over the last months and I think it's time he has
officially the right to approve changes in tripleo-validations.


+1



Consider this my +1 vote.

Martin

[1] 
http://stackalytics.com/?module=tripleo-validations=patches=pike

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo] Roadmap for Container CI work

[Jiří Stránský]

On 4.4.2017 22:01, Emilien Macchi wrote:

After our weekly meeting of today, I found useful to share and discuss
our roadmap for Container CI jobs in TripleO.
They are ordered by priority from the highest to lowest:

1. Swap ovb-nonha job with ovb-containers, enable introspection on the
container job and shuffle other coverage (e.g ssl) to other jobs
(HA?). It will help us to get coverage for ovb-containers scenario
again, without consuming more rh1 resources and keep existing
coverage.
2. Get multinode coverage of deployments - this should integrate with
the scenarios we already have defined for non-container deployment.
This is super important to cover all overcloud services, like we did
with classic deployments. It should be non voting to start and then
voting once it works. We should find a way to keep the same templates
as we have now, and just include the docker environment. In other
words, find a way to keep using:
https://github.com/openstack/tripleo-heat-templates/blob/master/ci/environments/scenario001-multinode.yaml
so we don't duplicate scenario environments.
3. Implement container upgrade job, which for Pike will be deploy a
baremetal overcloud, then migrate on upgrade to containers. Use
multinode jobs for this task. Start with a non-voting job and move to
the gate once it work. I also suggest to use scenarios framework, so
we keep good coverage.


The first iteration of this job is ready to be reviewed and landed. 
Please see the patches here [1].


The latest job execution didn't go all the way to success yet, it failed 
during Ansible upgrade steps execution [2], but i think the patches are 
now far enough that they would be good to merge anyway, and issues can 
be ironed out subsequently, as well as making the job actually 
Ocata->master rather than master->master (currently just switching from 
non-containers to containers).


[1] https://review.openstack.org/#/q/topic:container-upgrade
[2] 
http://logs.openstack.org/84/450784/8/experimental/gate-tripleo-ci-centos-7-containers-multinode-upgrades-nv/a1850f7/logs/undercloud/home/jenkins/overcloud_deploy.log.txt.gz



4. After we implement the workflow for minor updates, have a job with
tests container-to-container updates for minor (rolling) updates, this
ideally should add some coverage to ensure no downtime of APIs and
possibly checks for service restarts (ref recent bugs about bouncing
services on minor updates)
5. Once Pike is released and Queens starts, let's work on container to
containers upgrade job.

Any feedback or question is highly welcome,

Note: The proposal comes from shardy's notes on
https://etherpad.openstack.org/p/tripleo-container-ci - feel free to
contribute to the etherpad or mailing list.

Thanks,




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo] os-cloud-config retirement

[Jiří Stránský]

On 30.3.2017 17:39, Juan Antonio Osorio wrote:

Why not drive the post-config with something like shade over ansible?
Similar to what the kolla-ansible community is doing.


We could use those perhaps, if they bring enough benefit to add them to 
the container image(s) (i think we'd still want to drive it via a 
container rather than fully externally). It's quite tempting to just 
load a yaml file with the endpoint definitions and just iterate over 
them and let Ansible handle the actual API calls...


However, currently i can't see endpoint management in the cloud modules 
docs [1], just service management. Looks like there's still a feature 
gap at the moment.


Jirka

[1] http://docs.ansible.com/ansible/list_of_cloud_modules.html#openstack



On 30 Mar 2017 16:42, "Jiří Stránský" <ji...@redhat.com> wrote:


On 30.3.2017 14:58, Dan Prince wrote:


There is one case that I was thinking about reusing this piece of code
within a container to help initialize keystone endpoints. It would
require some changes and updates (to match how puppet-* configures
endpoints).

For TripleO containers we use various puppet modules (along with hiera)
to drive the creation of endpoints. This functionally works fine, but
is quite slow to execute (puppet is slow here) and takes several
minutes to complete. I'm wondering if a single optimized python script
might serve us better here. It could be driven via YAML (perhaps
similar to our Hiera), idempotent, and likely much faster than having
the code driven by puppet. This doesn't have to live in os-cloud-
config, but initially I thought that might be a reasonable place for
it. It is worth pointing out that this would be something that would
need to be driven by our t-h-t workflow and not a post-installation
task. So perhaps that makes it not a good fit for os-cloud-config. But
it is similar to the keystone initialization already there so I thought
I'd mention it.



I agree we could have an optimized python script instead of puppet to do
the init. However, os-cloud-config also doesn't strike me as the ideal
place.

What might be interesting is solving the keystone init within containers
along with our container entrypoint situation. We've talked earlier that we
may have to build our custom entrypoints into the images as we sometimes
need to do things that the current entrypoints don't seem fit for, or don't
give us enough control over what happens. This single optimized python
script for endpoint config you mentioned could be one of such in-image
entrypoint scripts. We could build multiple different scripts like this
into a single image and select the right one when starting the container
(defaulting to a script that handles the usual "worker" case, in this case
Keystone API).

This gets somewhat similar to the os-cloud-config usecase, but even if we
wanted a separate repo, or even a RPM for these, i suppose it would be
cleaner to just start from scratch rather than repurpose os-cloud-config.

Jirka



Dan

On Thu, 2017-03-30 at 08:13 -0400, Emilien Macchi wrote:


Hi,

os-cloud-config was deprecated in the Ocata release and is going to
be
removed in Pike.

TripleO project doesn't need it anymore and after some investigation
in codesearch.openstack.org, nobody is using it in OpenStack.
I'm working on the removal this cycle, please let us know any
concern.

Thanks,




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscrib
e
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev





__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo] os-cloud-config retirement

[Jiří Stránský]

On 30.3.2017 18:02, Bogdan Dobrelya wrote:

On 30.03.2017 15:40, Jiří Stránský wrote:


What might be interesting is solving the keystone init within containers
along with our container entrypoint situation. We've talked earlier that
we may have to build our custom entrypoints into the images as we
sometimes need to do things that the current entrypoints don't seem fit
for, or don't give us enough control over what happens. This single
optimized python script for endpoint config you mentioned could be one
of such in-image entrypoint scripts. We could build multiple different


I'm concerned of having entry points in-image. Could it be mounted as a
hostpath instead, then executed? Custom entry-points could replace
existing ones this way. This would allow keep kolla or other images
clean from side changes.


That was actually my initial thought as well, but it means more 
entanglement between the containers and the bare-metal hosts, and 
creates some new issues.


E.g. it makes container image versioning harder. We'd need to implement 
additional logic to make sure we use the correct entrypoint version for 
a particular container image version (think rolling back to an older 
image but still using the newest entrypoint, perhaps those two not being 
fully compatible, and having the container crash because of this). This 
alone is quite disadvantageous IMO.


Jirka




scripts like this into a single image and select the right one when
starting the container (defaulting to a script that handles the usual


We could use a clean container and mount in that we need. Those entry
points looks similar to heat agent hooks, right? I think they should be
packaged as a separate artifacts.


"worker" case, in this case Keystone API).

This gets somewhat similar to the os-cloud-config usecase, but even if
we wanted a separate repo, or even a RPM for these, i suppose it would
be cleaner to just start from scratch rather than repurpose
os-cloud-config.

Jirka






__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo] os-cloud-config retirement

[Jiří Stránský]

On 30.3.2017 14:58, Dan Prince wrote:

There is one case that I was thinking about reusing this piece of code
within a container to help initialize keystone endpoints. It would
require some changes and updates (to match how puppet-* configures
endpoints).

For TripleO containers we use various puppet modules (along with hiera)
to drive the creation of endpoints. This functionally works fine, but
is quite slow to execute (puppet is slow here) and takes several
minutes to complete. I'm wondering if a single optimized python script
might serve us better here. It could be driven via YAML (perhaps
similar to our Hiera), idempotent, and likely much faster than having
the code driven by puppet. This doesn't have to live in os-cloud-
config, but initially I thought that might be a reasonable place for
it. It is worth pointing out that this would be something that would
need to be driven by our t-h-t workflow and not a post-installation
task. So perhaps that makes it not a good fit for os-cloud-config. But
it is similar to the keystone initialization already there so I thought
I'd mention it.


I agree we could have an optimized python script instead of puppet to do 
the init. However, os-cloud-config also doesn't strike me as the ideal 
place.


What might be interesting is solving the keystone init within containers 
along with our container entrypoint situation. We've talked earlier that 
we may have to build our custom entrypoints into the images as we 
sometimes need to do things that the current entrypoints don't seem fit 
for, or don't give us enough control over what happens. This single 
optimized python script for endpoint config you mentioned could be one 
of such in-image entrypoint scripts. We could build multiple different 
scripts like this into a single image and select the right one when 
starting the container (defaulting to a script that handles the usual 
"worker" case, in this case Keystone API).


This gets somewhat similar to the os-cloud-config usecase, but even if 
we wanted a separate repo, or even a RPM for these, i suppose it would 
be cleaner to just start from scratch rather than repurpose os-cloud-config.


Jirka



Dan

On Thu, 2017-03-30 at 08:13 -0400, Emilien Macchi wrote:

Hi,

os-cloud-config was deprecated in the Ocata release and is going to
be
removed in Pike.

TripleO project doesn't need it anymore and after some investigation
in codesearch.openstack.org, nobody is using it in OpenStack.
I'm working on the removal this cycle, please let us know any
concern.

Thanks,


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo] propose Alex Schultz core on tripleo-heat-templates

[Jiří Stránský]

On 13.3.2017 15:30, Emilien Macchi wrote:

Hi,

Alex is already core on instack-undercloud and puppet-tripleo.
His involvement and knowledge in TripleO Heat Templates has been very
appreciated over the last months and I think we can give him +2 on
this project.

As usual, feel free to vote -1/+1 on this proposal.


+1



Thanks,




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo] Fwd: TripleO mascot - how can I help your team?

[Jiří Stránský]

On 10.3.2017 17:26, Heidi Joy Tretheway wrote:

Hi TripleO team,

Here’s an update on your project logo. Our illustrator tried to be as true as
possible to your original, while ensuring it matched the line weight, color
palette and style of the rest. We also worked to make sure that three Os in the
logo are preserved. Thanks for your patience as we worked on this! Feel free to
direct feedback to me.


I think it's great! The connection to our current logo is pretty clear 
to me, so hopefully we wouldn't be confusing anyone too much by 
switching to the new logo. Thanks for the effort!


Also personally i like the color scheme change to a more 
playful/cartoony look as you mentioned in your other e-mail.


Jirka





__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO] Proposing Dmitry Tantsur and Alex Schultz as instack-undercloud cores

[Jiří Stránský]

On 31.1.2017 19:08, Ben Nemec wrote:



On 01/31/2017 11:03 AM, James Slagle wrote:

On Tue, Jan 31, 2017 at 11:02 AM, Ben Nemec  wrote:

In the spirit of all the core team changes, here are a couple more I'd like
to propose.

Dmitry has been very helpful reviewing in instack-undercloud for a long time
so this is way overdue.  I'm also going to propose that he be able to +2
anything Ironic-related in TripleO since that is his primary area of
expertise.

Alex has ramped up quickly on TripleO and has also been helping out with
instack-undercloud quite a bit.  He's already core for the puppet modules,
and a lot of the changes to instack-undercloud these days are primarily in
the puppet manifest so it's not a huge stretch to add him.

As usual, TripleO cores please vote and/or provide comments.  Thanks.


+1 both



-Ben

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


+1. Dmitry is also one of the top reviewers and committers to
tripleo-docs. I wouldn't be opposed to him having +2 there as well.



Oh, good call.  I forgot to mention that too.  +1 to adding him as docs
core as well.


+1

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO] Proposing Sergey (Sagi) Shnaidman for core on tripleo-ci

[Jiří Stránský]

On 24.1.2017 18:03, Juan Antonio Osorio wrote:

Sagi (sshnaidm on IRC) has done significant work in TripleO CI (both
on the current CI solution and in getting tripleo-quickstart jobs for
it); So I would like to propose him as part of the TripleO CI core team.



+1


I think he'll make a great addition to the team and will help move CI
issues forward quicker.

Best Regards,





__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo] Update TripleO core members

[Jiří Stránský]

On 23.1.2017 20:03, Emilien Macchi wrote:

Greeting folks,

I would like to propose some changes in our core members:

- Remove Jay Dobies who has not been active in TripleO for a while
(thanks Jay for your hard work!).
- Add Flavio Percoco core on tripleo-common and tripleo-heat-templates
docker bits.
- Add Steve Backer on os-collect-config and also docker bits in
tripleo-common and tripleo-heat-templates.

Indeed, both Flavio and Steve have been involved in deploying TripleO
in containers, their contributions are very valuable. I would like to
encourage them to keep doing more reviews in and out container bits.

As usual, core members are welcome to vote on the changes.


+1



Thanks,




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo] Proposing Honza Pokorny core on tripleo-ui

[Jiří Stránský]

On 24.1.2017 14:52, Emilien Macchi wrote:

I have been discussed with TripleO UI core reviewers and it's pretty
clear Honza's work has been valuable so we can propose him part of
Tripleo UI core team.
His quality of code and reviews make him a good candidate and it would
also help the other 2 core reviewers to accelerate the review process
in UI component.

Like usual, this is open for discussion, Tripleo UI core and TripleO
core, please vote.


+1



Thanks,




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo] Proposing Alex Schultz core on puppet-tripleo

[Jiří Stránský]

On 1.12.2016 23:26, Emilien Macchi wrote:

Team,

Alex Schultz (mwhahaha on IRC) has been active on TripleO since a few
months now.  While he's very active in different areas of TripleO, his
reviews and contributions on puppet-tripleo have been very useful.
Alex is a Puppet guy and also the current PTL of Puppet OpenStack. I
think he perfectly understands how puppet-tripleo works. His
involvement in the project and contributions on puppet-tripleo deserve
that we allow him to +2 puppet-tripleo.

Thanks Alex for your involvement and hard work in the project, this is
very appreciated!

As usual, I'll let the team to vote about this proposal.

Thanks,


+1

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo] proposing Michele Baldessari part of core team

[Jiří Stránský]

+1, Michele does great reviews, and his contributions around HA and 
upgrades have been crucial.


On 4.11.2016 18:40, Emilien Macchi wrote:

MIchele Baldessari (bandini on IRC) has consistently demonstrated high
levels of contributions in TripleO projects, specifically in High
Availability area where's he's for us a guru (I still don't understand
how pacemaker works, but hopefully he does).

He has done incredible work on composable services and also on
improving our HA configuration by following reference architectures.
Always here during meetings, and on #tripleo to give support to our
team, he's a great team player and we are lucky to have him onboard.
I believe he would be a great core reviewer on HA-related work and we
expect his review stats to continue improving as his scope broadens
over time.

As usual, feedback is welcome and please vote for this proposal!

Thanks,




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [heat][yaql] Deep merge map of lists?

[Jiří Stránský]

On 30.8.2016 18:16, Zane Bitter wrote:

On 30/08/16 12:02, Steven Hardy wrote:

  debug_tripleo2:
value:
  yaql:
expression: $.data.l.reduce($1.mergeWith($2))
data:
  l:
- "gnocchi_metricd_node_names": ["overcloud-controller-0",
  "overcloud-controller-1", "overcloud-controller-2"]
  "tripleo_packages_node_names": ["overcloud-controller-0", 
"overcloud-controller-1", "overcloud-controller-2"]
- "nova_compute_node_names": ["overcloud-compute-0"]
  "tripleo_packages_node_names": ["overcloud-compute-0"]
  "tripleo_packages_node_names2": ["overcloud-compute-0"]
- "ceph_osd_node_names": ["overcloud-cephstorage-0"]
  "tripleo_packages_node_names": ["overcloud-cephstorage-0"]
  "tripleo_packages_node_names2": ["overcloud-cephstorage-0"]

$ heat output-show foo5 debug_tripleo2
stack output show" instead
Output error: can only concatenate tuple (not "list") to tuple

I've not dug too deeply yet, but assuming that's a yaql error vs a heat bug
it looks like it won't work.


It works flawlessly in yaqluator, so that sounds like a Heat bug.


Ack, i reported it so that it doesn't fall through the cracks:

https://bugs.launchpad.net/heat/+bug/1618538


Jirka



- ZB

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [heat][yaql] Deep merge map of lists?

[Jiří Stránský]

On 30.8.2016 18:02, Steven Hardy wrote:

On Tue, Aug 30, 2016 at 04:10:47PM +0200, Jiří Stránský wrote:


On 30.8.2016 10:17, Steven Hardy wrote:




Yeah, that gets us closer, but we do need to handle more than one value
(list entry) per key, e.g:

 data:
   l:
 - "gnocchi_metricd_node_names": ["a0", "a1", "a2"]
   "tripleo_packages_node_names": ["a0", "a1", "a2"]
 - "nova_compute_node_names": ["b0"]
   "tripleo_packages_node_names": ["b0"]

Output needs to be like:

 "gnocchi_metricd_node_names": ["a0", "a1", "a2"]
 "tripleo_packages_node_names": ["a0", "a1", "a2", "b0"]
 "nova_compute_node_names": ["b0"]



Hoping this could do it:

[stack@instack ~]$ cat yaq.yaml
heat_template_version: 2016-10-14

outputs:
  debug:
value:
  yaql:
expression: $.data.l.reduce($1.mergeWith($2))
data:
  l:
- "gnocchi_metricd_node_names": ["a0", "a1", "a2"]
  "tripleo_packages_node_names": ["a0", "a1", "a2"]
- "nova_compute_node_names": ["b0"]
  "tripleo_packages_node_names": ["b0"]


Thanks for this!

Unfortunately I dont think it works with more than two list items:

  debug_tripleo2:
value:
  yaql:
expression: $.data.l.reduce($1.mergeWith($2))
data:
  l:
- "gnocchi_metricd_node_names": ["overcloud-controller-0",
  "overcloud-controller-1", "overcloud-controller-2"]
  "tripleo_packages_node_names": ["overcloud-controller-0", 
"overcloud-controller-1", "overcloud-controller-2"]
- "nova_compute_node_names": ["overcloud-compute-0"]
  "tripleo_packages_node_names": ["overcloud-compute-0"]
  "tripleo_packages_node_names2": ["overcloud-compute-0"]
- "ceph_osd_node_names": ["overcloud-cephstorage-0"]
  "tripleo_packages_node_names": ["overcloud-cephstorage-0"]
  "tripleo_packages_node_names2": ["overcloud-cephstorage-0"]

$ heat output-show foo5 debug_tripleo2
stack output show" instead
Output error: can only concatenate tuple (not "list") to tuple

I've not dug too deeply yet, but assuming that's a yaql error vs a heat bug
it looks like it won't work.


Hmm yea that's strange, because YAQL has a test case for reduce() with 5 
items:


https://github.com/openstack/yaql/blob/f71a0305089997cbfa5ff00f660920711b04f39e/yaql/tests/test_queries.py#L337-L339

Anyway, good that we have the solution below that works :)

Jirka



However I did find an approach earler with therve which seems to do what is
needed:

 debug_tripleo:
value:
  yaql:
# $.selectMany($.items()).groupBy($[0], $[1][0])
# reduce($1 + $2)')
# dict($.selectMany($.items()).groupBy($[0], $[1], [$[0],
# $[1].flatten()]))
expression: dict($.data.l.selectMany($.items()).groupBy($[0], $[1],
[$[0], $[1].flatten()]))
data:
  l:
- "gnocchi_metricd_node_names": ["overcloud-controller-0",
  "overcloud-controller-1", "overcloud-controller-2"]
  "tripleo_packages_node_names": ["overcloud-controller-0", 
"overcloud-controller-1", "overcloud-controller-2"]
  "tripleo_packages_node_names2": ["overcloud-controller-0", 
"overcloud-controller-1", "overcloud-controller-2"]
- "nova_compute_node_names": ["overcloud-compute-0"]
  "tripleo_packages_node_names": ["overcloud-compute-0"]
  "tripleo_packages_node_names2": ["overcloud-compute-0"]
- "ceph_osd_node_names": ["overcloud-cephstorage-0"]
  "tripleo_packages_node_names": ["overcloud-cephstorage-0"]
  "tripleo_packages_node_names2": ["overcloud-cephstorage-0"]

Output:

$ heat output-show foo5 debug_tripleo
stack output show" instead
{
  "gnocchi_metricd_node_names": [
"overcloud-controller-0",
"overcloud-controller-1",
"overcloud-controller-2"
  ],
  "tripleo_packages_node_names": [
"overcloud-controller-0",
"overcloud

Re: [openstack-dev] [heat][yaql] Deep merge map of lists?

[Jiří Stránský]

 expression: $.data.l.reduce($1.mergeWith($2))


Or maybe it's better with seed value for reduce, just in case:

$.data.l.reduce($1.mergeWith($2), {})


Jirka

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [heat][yaql] Deep merge map of lists?

[Jiří Stránský]

On 30.8.2016 10:17, Steven Hardy wrote:




Yeah, that gets us closer, but we do need to handle more than one value
(list entry) per key, e.g:

 data:
   l:
 - "gnocchi_metricd_node_names": ["a0", "a1", "a2"]
   "tripleo_packages_node_names": ["a0", "a1", "a2"]
 - "nova_compute_node_names": ["b0"]
   "tripleo_packages_node_names": ["b0"]

Output needs to be like:

 "gnocchi_metricd_node_names": ["a0", "a1", "a2"]
 "tripleo_packages_node_names": ["a0", "a1", "a2", "b0"]
 "nova_compute_node_names": ["b0"]



Hoping this could do it:

[stack@instack ~]$ cat yaq.yaml
heat_template_version: 2016-10-14

outputs:
  debug:
value:
  yaql:
expression: $.data.l.reduce($1.mergeWith($2))
data:
  l:
- "gnocchi_metricd_node_names": ["a0", "a1", "a2"]
  "tripleo_packages_node_names": ["a0", "a1", "a2"]
- "nova_compute_node_names": ["b0"]
  "tripleo_packages_node_names": ["b0"]


[stack@instack ~]$ heat output-show yaq debug
WARNING (shell) "heat output-show" is deprecated, please use "openstack 
stack output show" instead

{
  "gnocchi_metricd_node_names": [
"a0",
"a1",
"a2"
  ],
  "tripleo_packages_node_names": [
"a0",
"a1",
"a2",
"b0"
  ],
  "nova_compute_node_names": [
"b0"
  ]
}

Jirka

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo] State of upgrade CLI commands

[Jiří Stránský]

On 16.8.2016 21:08, Brad P. Crochet wrote:

Hello TripleO-ians,

I've started to look again at the introduced, but unused/undocumented
upgrade commands. It seems to me that given the current state of the
upgrade process (at least from Liberty -> Mitaka), these commands make
a lot less sense.

I see one of two directions to take on this. Of course I would love to
hear other options.

1) Revert these commands immediately, and forget they ever existed.
They don't exactly work, and as I said, were never officially
documented, so I don't think a revert is out of the question.

or

2) Do a major overhaul, and rethink the interface entirely. For
instance, the L->M upgrade introduced a couple of new steps (the AODH
migration and the Keystone migration). These would have either had to
have completely new commands added, or have some type of override to
the existing upgrade command to handle them.

Personally, I would go for step 1. The 'overcloud deploy' command can
accomplish all of the upgrade steps that involve Heat. In order for
the new upgrade commands to work properly, there's a lot that needs to
be refactored out of the deploy command itself so that it can be
shared with deploy and upgrade, like passing of passwords and the
like. I just don't see a need for discrete commands when we have an
existing command that will do it for us. And with the addition of an
answer file, it makes it even easier.

Thoughts?



+1 for approach no. 1. Currently `overcloud deploy` meets the upgrade 
needs and it gave us some flexibility to e.g. do migrations like AODH 
and Keystone WSGI. I don't think we should have a special command for 
upgrades at this point.


The situation may change as we go towards upgrades of composable 
services, and perhaps wrap upgrades in Mistral if/when applicable, but 
then the potential upgrade command(s) would probably be different from 
the current ones anyway, so +1 for removing them.


Jirka

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO] Aodh upgrades - Request backport exception for stable/liberty

[Jiří Stránský]

On 16.5.2016 23:54, Pradeep Kilambi wrote:

On Mon, May 16, 2016 at 3:33 PM, James Slagle 
wrote:


On Mon, May 16, 2016 at 10:34 AM, Pradeep Kilambi  wrote:

Hi Everyone:

I wanted to start a discussion around considering backporting Aodh to
stable/liberty for upgrades. We have been discussing quite a bit on whats
the best way for our users to upgrade ceilometer alarms to Aodh when

moving

from liberty to mitaka. A quick refresh on what changed, In Mitaka,
ceilometer alarms were replaced by Aodh. So only way to get alarms
functionality is use aodh. Now when the user kicks off upgrades from

liberty

to Mitaka, we want to make sure alarms continue to function as expected
during the process which could take multiple days. To accomplish this I
propose the following approach:

* Backport Aodh functionality to stable/liberty. Note, Aodh

functionality is

backwards compatible, so with Aodh running, ceilometer api and client

will

redirect requests to Aodh api. So this should not impact existing users

who

are using ceilometer api or client.

* As part of Aodh deployed via heat stack update, ceilometer alarms

services

will be replaced by openstack-aodh-*. This will be done by the puppet

apply

as part of stack convergence phase.

* Add checks in the Mitaka pre upgrade steps when overcloud install kicks
off to check and warn the user to update to liberty + aodh to ensure

aodh is

running. This will ensure heat stack update is run and, if alarming is

used,

Aodh is running as expected.

The upgrade scenarios between various releases would work as follows:

Liberty -> Mitaka

* Upgrade starts with ceilometer alarms running
* A pre-flight check will kick in to make sure Liberty is upgraded to
liberty + aodh with stack update
* Run heat stack update to upgrade to aodh
* Now ceilometer alarms should be removed and Aodh should be running
* Proceed with mitaka upgrade
* End result, Aodh continue to run as expected

Liberty + aodh -> Mitaka:

* Upgrade starts with Aodh running
* A pre-flight check will kick in to make sure Liberty is upgraded to

Aodh

with stack update
* Confirming Aodh is indeed running, proceed with Mitaka upgrade with

Aodh

running
* End result, Aodh continue to be run as expected


This seems to be a good way to get the upgrades working for aodh. Other

less

effective options I can think of are:

1. Let the Mitaka upgrade kick off and do "yum update" which replace aodh
during migration, alarm functionality will be down until puppet converge
runs and configures Aodh. This means alarms will be down during upgrade
which is not ideal.

2. During Mitaka upgrades, replace with Aodh and add a bash script that
fully configures Aodh and ensures aodh is functioning. This will involve
significant work and results in duplicating everything puppet does today.


How much duplication would this really be? Why would it have to be in bash?



Well pretty much entire aodh configuration will need to happen, Here is
what we do in devstack, something along these lines[1]. So in short, we'll
need to install, create users, configure db and coordination backends,
configure api to run under mod wsgi. Sure, it doesn't have to be bash,
assumed that would be easiest to invoke during upgrades.





Could it be:

Liberty -> Mitaka

* Upgrade starts with ceilometer alarms running
* Add a new hook for the first step of Mitaka upgrade that does:
  ** sets up mitaka repos
  ** migrates from ceilometer alarms to aodh, can use puppet
  ** ensures aodh is running
* Proceed with rest of mitaka upgrade

At most, it seems we'd have to surround the puppet apply with some
pacemaker commands to possibly set maintenance mode and migrate
constraints.

The puppet manifest itself would just be the includes and classes for aodh.




Yea I guess we could do something like this, i'm not fully clear on the
details on how and when this would be called. But with the below caveat you
mentioned already.



Yes this is a possibility, it's still not fully utilizing the Puppet we 
have for deployment, we'd have at least a custom manifest, but hopefully 
it wouldn't be too big.


In case the AODH classes include some other Puppet classes from their 
code, we could end up applying more config changes than desired in this 
phase and break something. I'm hoping that this is more of a theoretical 
concern rather than practical, but probably deserves some verification.







One complication might be that the aodh packages from Mitaka might
pull in new deps that required updating other OpenStack services,
which we wouldn't yet want to do. That is probably worth confirming
though.



Yea we will be pulling in at least some new oslo deps and client libraries
for sure. But wouldnt yum update during the upgrades do that anyway? or
would aodh setup run before yum update phase of upgrade process?


Good question :) We could probably also do it in the middle of 
controller update phase, between step 1 (stop 

Re: [openstack-dev] [TripleO] propose ejuaso for core

[Jiří Stránský]

+1

On 14.3.2016 15:38, Dan Prince wrote:

http://russellbryant.net/openstack-stats/tripleo-reviewers-180.txt

Our top reviewer over the last half a year ejuaso (goes by Ozz for
Osorio or jaosorior on IRC). His reviews seem consistent, he
consistently attends the meetings and he chimes in on lots of things.
I'd like to propose we add him to our core team (probably long overdue
now too).

If you agree please +1. If there is no negative feedback I'll add him
next Monday.

Dan

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO] core members for tripleo-ui

[Jiří Stránský]

+1

On 29.2.2016 16:27, Dan Prince wrote:

There is a new projects for the ui called tripleo-ui. As most of the
existing TripleO core members aren't going to be reviewing UI specific
patches is seems reasonable that we might add a few review candidates
who can focus specifically on UI specific patches.

I'd like to proposed we add Jiri Tomasek and Ana Krivokapic as core
candidates that will focus primarily on the UI. They would be added to
tripleo core but would agree to only +2 patches within the UI for now,
or at least until they are re-nominated for more general TripleO core,
etc.

Core members if you could please vote on this so we can add these
members at the close of this week. Thanks,

Dan

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO] Should we have a TripleO API, or simply use Mistral?

[Jiří Stránský]

On 18.1.2016 19:49, Tzu-Mainn Chen wrote:

- Original Message -

On Thu, 2016-01-14 at 16:04 -0500, Tzu-Mainn Chen wrote:


- Original Message -

On Wed, Jan 13, 2016 at 04:41:28AM -0500, Tzu-Mainn Chen wrote:

Hey all,

I realize now from the title of the other TripleO/Mistral thread
[1] that
the discussion there may have gotten confused.  I think using
Mistral for
TripleO processes that are obviously workflows - stack
deployment, node
registration - makes perfect sense.  That thread is exploring
practicalities
for doing that, and I think that's great work.

What I inappropriately started to address in that thread was a
somewhat
orthogonal point that Dan asked in his original email, namely:

"what it might look like if we were to use Mistral as a
replacement for the
TripleO API entirely"

I'd like to create this thread to talk about that; more of a
'should we'
than 'can we'.  And to do that, I want to indulge in a thought
exercise
stemming from an IRC discussion with Dan and others.  All, please
correct
me
if I've misstated anything.

The IRC discussion revolved around one use case: deploying a Heat
stack
directly from a Swift container.  With an updated patch, the Heat
CLI can
support this functionality natively.  Then we don't need a
TripleO API; we
can use Mistral to access that functionality, and we're done,
with no need
for additional code within TripleO.  And, as I understand it,
that's the
true motivation for using Mistral instead of a TripleO API:
avoiding custom
code within TripleO.

That's definitely a worthy goal... except from my perspective,
the story
doesn't quite end there.  A GUI needs additional functionality,
which boils
down to: understanding the Heat deployment templates in order to
provide
options for a user; and persisting those options within a Heat
environment
file.

Right away I think we hit a problem.  Where does the code for
'understanding
options' go?  Much of that understanding comes from the
capabilities map
in tripleo-heat-templates [2]; it would make sense to me that
responsibility
for that would fall to a TripleO library.

Still, perhaps we can limit the amount of TripleO code.  So to
give API
access to 'getDeploymentOptions', we can create a Mistral
workflow.

   Retrieve Heat templates from Swift -> Parse capabilities map

Which is fine-ish, except from an architectural perspective
'getDeploymentOptions' violates the abstraction layer between
storage and
business logic, a problem that is compounded because
'getDeploymentOptions'
is not the only functionality that accesses the Heat templates
and needs
exposure through an API.  And, as has been discussed on a
separate TripleO
thread, we're not even sure Swift is sufficient for our needs;
one possible
consideration right now is allowing deployment from templates
stored in
multiple places, such as the file system or git.


Actually, that whole capabilities map thing is a workaround for a
missing
feature in Heat, which I have proposed, but am having a hard time
reaching
consensus on within the Heat community:

https://review.openstack.org/#/c/196656/

Given that is a large part of what's anticipated to be provided by
the
proposed TripleO API, I'd welcome feedback and collaboration so we
can move
that forward, vs solving only for TripleO.


Are we going to have duplicate 'getDeploymentOptions' workflows
for each
storage mechanism?  If we consolidate the storage code within a
TripleO
library, do we really need a *workflow* to call a single
function?  Is a
thin TripleO API that contains no additional business logic
really so bad
at that point?


Actually, this is an argument for making the validation part of the
deployment a workflow - then the interface with the storage
mechanism
becomes more easily pluggable vs baked into an opaque-to-operators
API.

E.g, in the long term, imagine the capabilities feature exists in
Heat, you
then have a pre-deployment workflow that looks something like:

1. Retrieve golden templates from a template store
2. Pass templates to Heat, get capabilities map which defines
features user
must/may select.
3. Prompt user for input to select required capabilites
4. Pass user input to Heat, validate the configuration, get a
mapping of
required options for the selected capabilities (nested validation)
5. Push the validated pieces ("plan" in TripleO API terminology) to
a
template store

This is a pre-deployment validation workflow, and it's a superset
of the
getDeploymentOptions feature you refer to.

Historically, TripleO has had a major gap wrt workflow, meaning
that we've
always implemented it either via shell scripts (tripleo-incubator)
or
python code (tripleo-common/tripleo-client, potentially TripleO
API).

So I think what Dan is exploring is, how do we avoid reimplementing
a
workflow engine, when a project exists which already does that.


My gut reaction is to say that proposing Mistral in place of a
TripleO API
is to look at the engineering concerns from the wrong
direction.  The

Re: [openstack-dev] [TripleO] Deploy Overcloud Keystone in HTTPD

[Jiří Stránský]

On 19.1.2016 03:59, Adam Young wrote:

I have a review here for switching Keystone to HTTPD

https://review.openstack.org/#/c/269377/

But I have no idea how to kick off the CI to really test it.  The check
came back way too quick for it to have done a full install; less than 3
minutes.  I think it was little more than a lint check.

How can I get a real sense of if it is this easy or if there is
something more that needs to be done?


Jenkins reports in two phases, first come the unit tests (in minutes), 
then the integration tests (in about 1.5 hrs minimum, depending on the 
CI load).


Jirka

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev