[rdo-users] download-logs.sh feature for zuul logs

[Wesley Hayutin]

Greetings,

RFE to rdo projects zuul and others... would be to deploy the feature that
includes the option to download logs from the log server. Example is here
[1]

Thanks all!

[1]
https://6d6df89134025ef5b0e9-648722ac87374da2f576895eac8df5a8.ssl.cf5.rackcdn.com/794592/4/check/tripleo-ci-centos-8-standalone/7e2d1d6/download-logs.sh
___
users mailing list
users@lists.rdoproject.org
http://lists.rdoproject.org/mailman/listinfo/users

To unsubscribe: users-unsubscr...@lists.rdoproject.org

[rdo-dev] FYI.. docker.io rate limiting

[Wesley Hayutin]

Greetings,

Most of the tripleo ci jobs at
https://ci.centos.org/view/rdo/view/promotion-pipeline/
are now failing due to docker.io rate limits.  This may impact how often
the various releases promote.

0/
___
dev mailing list
dev@lists.rdoproject.org
http://lists.rdoproject.org/mailman/listinfo/dev

To unsubscribe: dev-unsubscr...@lists.rdoproject.org

Re: [rdo-dev] Planned outage of review.rdoproject.org: 2021-03-15 from 14:00 to 16:00 UTC

[Wesley Hayutin]

Thanks Nicolas!


On Mon, Mar 15, 2021 at 9:51 AM Nicolas Hicher  wrote:

> Hello folks,
>
> The upgrade operation is done, all services are up and running.
>
> New user interfaces have been deployed.
>
> Gerrit has been updated to the last stable version.
>
> You may have to clean your browser cache if you get 'Plugin install error:
> TypeError: self.onAction ... delete-project.js' on gerrit page.
>
> Regards,
> Nicolas , on behalf of the Software Factory Operation Team
>
> [1] https://phabricator.wikimedia.org/T256560
>
>
> On 3/9/21 1:48 PM, nhic...@redhat.com wrote:
>
> Hello folks,
>
> We plan to upgrade the software factory deployment on 2021-03-15 from 14:00 
> to 16:00 UTC to the next 3.6 release(including Gerrit 3.2, Zuul and Nodepool 
> 4.0 and a new web-ui for the welcome page of software-factory).
>
> Service interruption is expected, including:
> - Zuul CI not running jobs for gerrit, github or opendev.
> - RDO Trunk not building new packages.
> - DLRN API.
> - review.rdoproject.org and softwarefactory-project.io gerrit service.
> - www.rdoproject.org and lists.rdoproject.org.
>
> We expect that the interruption of services will be less than 2 hours.
>
> Regards,
> Nicolas , on behalf of the Software Factory Operation Team
>
> ___
> dev mailing list
> dev@lists.rdoproject.org
> http://lists.rdoproject.org/mailman/listinfo/dev
>
> To unsubscribe: dev-unsubscr...@lists.rdoproject.org
>
___
dev mailing list
dev@lists.rdoproject.org
http://lists.rdoproject.org/mailman/listinfo/dev

To unsubscribe: dev-unsubscr...@lists.rdoproject.org

Re: [rdo-dev] [rdo-users] [RDO] Weekly Status for 2020-10-09

[Wesley Hayutin]

On Mon, Oct 12, 2020 at 4:59 AM YATIN KAREL  wrote:

> Promotions:-
>   * Latest promotions (TripleO CI):
> * Master: 06th Oct
> * Ussuri: 09th Oct
> * Train(C8): 08th Oct
> * Train(C7): 17th Sept
> * Stein: 08th Oct
>   * Know blocker master
> * https://bugs.launchpad.net/tripleo/+bug/1897947 (octavia component)
> * https://bugs.launchpad.net/tripleo/+bug/1898931 (integration)
>   * Known blocker train(C7)
> * https://review.rdoproject.org/r/#/c/29397/
>
> Packages
>   * Tempest is bumped to 25.0.0 in Ussuri and Train
>   * Octavia-tempest-plugin is pinned in victoria+ due to missing
> deps(httpx)
> * https://review.rdoproject.org/etherpad/p/httpx-centos8-octavia
>
> Victoria Release Preparation
>   * https://review.rdoproject.org/etherpad/p/victoria-release-preparation
>   * Most of the upstream projects have released RC2 bits, all of these
> projects are tagged for release and available in CentOS Mirrors
> * http://mirror.centos.org/centos/8/cloud/x86_64/openstack-victoria/
>   * https://review.rdoproject.org/r/#/q/topic:victoria-branching
>   * openstack/requirements has cut stable/victoria, and now master is
> switched to wallaby tags in rdoinfo
> * https://review.rdoproject.org/r/29934
>   * Victoria jobs are being added by TripleO CI Team
> * https://hackmd.io/sasvoKjTSkabGlFOTkK_kw#bugs-and-reviews
>

Just a quick update here... still a WIP, do not use Victoria for tripleo
deployments.

We did have our first promotion today, /me notes this is NOT with full
coverage yet.
https://trunk.rdoproject.org/api-centos8-victoria/api/civotes_agg_detail.html?ref_hash=13431bf4c5a8ee8c0231f11d43050939
https://trunk.rdoproject.org/centos8-victoria/current-tripleo/
http://images.rdoproject.org/centos8/victoria/rdo_trunk/
https://hub.docker.com/u/tripleovictoria

Nice progress though :)





>
> Other
>   * Implementation of sources verification with GPG signature is on
> progress:
> *
> https://review.rdoproject.org/r/#/q/topic:gpg-verification+(status:open+OR+status:merged)
>
>
> On behalf of RDO
> ___
> users mailing list
> us...@lists.rdoproject.org
> http://lists.rdoproject.org/mailman/listinfo/users
>
> To unsubscribe: users-unsubscr...@lists.rdoproject.org
>
>
>
>
>
>
> ___
> dev mailing list
> dev@lists.rdoproject.org
> http://lists.rdoproject.org/mailman/listinfo/dev
>
> To unsubscribe: dev-unsubscr...@lists.rdoproject.org
>
___
dev mailing list
dev@lists.rdoproject.org
http://lists.rdoproject.org/mailman/listinfo/dev

To unsubscribe: dev-unsubscr...@lists.rdoproject.org

Re: [rdo-users] [rdo-dev] [RDO] Weekly Status for 2020-10-09

[Wesley Hayutin]

On Mon, Oct 12, 2020 at 4:59 AM YATIN KAREL  wrote:

> Promotions:-
>   * Latest promotions (TripleO CI):
> * Master: 06th Oct
> * Ussuri: 09th Oct
> * Train(C8): 08th Oct
> * Train(C7): 17th Sept
> * Stein: 08th Oct
>   * Know blocker master
> * https://bugs.launchpad.net/tripleo/+bug/1897947 (octavia component)
> * https://bugs.launchpad.net/tripleo/+bug/1898931 (integration)
>   * Known blocker train(C7)
> * https://review.rdoproject.org/r/#/c/29397/
>
> Packages
>   * Tempest is bumped to 25.0.0 in Ussuri and Train
>   * Octavia-tempest-plugin is pinned in victoria+ due to missing
> deps(httpx)
> * https://review.rdoproject.org/etherpad/p/httpx-centos8-octavia
>
> Victoria Release Preparation
>   * https://review.rdoproject.org/etherpad/p/victoria-release-preparation
>   * Most of the upstream projects have released RC2 bits, all of these
> projects are tagged for release and available in CentOS Mirrors
> * http://mirror.centos.org/centos/8/cloud/x86_64/openstack-victoria/
>   * https://review.rdoproject.org/r/#/q/topic:victoria-branching
>   * openstack/requirements has cut stable/victoria, and now master is
> switched to wallaby tags in rdoinfo
> * https://review.rdoproject.org/r/29934
>   * Victoria jobs are being added by TripleO CI Team
> * https://hackmd.io/sasvoKjTSkabGlFOTkK_kw#bugs-and-reviews
>

Just a quick update here... still a WIP, do not use Victoria for tripleo
deployments.

We did have our first promotion today, /me notes this is NOT with full
coverage yet.
https://trunk.rdoproject.org/api-centos8-victoria/api/civotes_agg_detail.html?ref_hash=13431bf4c5a8ee8c0231f11d43050939
https://trunk.rdoproject.org/centos8-victoria/current-tripleo/
http://images.rdoproject.org/centos8/victoria/rdo_trunk/
https://hub.docker.com/u/tripleovictoria

Nice progress though :)





>
> Other
>   * Implementation of sources verification with GPG signature is on
> progress:
> *
> https://review.rdoproject.org/r/#/q/topic:gpg-verification+(status:open+OR+status:merged)
>
>
> On behalf of RDO
> ___
> users mailing list
> users@lists.rdoproject.org
> http://lists.rdoproject.org/mailman/listinfo/users
>
> To unsubscribe: users-unsubscr...@lists.rdoproject.org
>
>
>
>
>
>
> ___
> dev mailing list
> d...@lists.rdoproject.org
> http://lists.rdoproject.org/mailman/listinfo/dev
>
> To unsubscribe: dev-unsubscr...@lists.rdoproject.org
>
___
users mailing list
users@lists.rdoproject.org
http://lists.rdoproject.org/mailman/listinfo/users

To unsubscribe: users-unsubscr...@lists.rdoproject.org

Re: [rdo-users] [TripleO][Ussuri] image prepare, cannot download containers image prepare recently

[Wesley Hayutin]

On Mon, Aug 31, 2020 at 1:23 PM Ruslanas Gžibovskis 
wrote:

> Hi all,
>
> I have noticed, that recently my undercloud is not able to download images
> [0]. I have provided newly generated containers-prepare-parameter.yaml and
> outputs from container image prepare
>
> providing --verbose and later beginning of --debug (in the end) [0]
>
> were there any changes? As "openstack tripleo container image prepare
> default --output-env-file containers-prepare-parameter.yaml
> --local-push-destination" have prepared a bit different file, compared
> what was previously: NEW # namespace: docker.io/tripleou VS namespace:
> docker.io/tripleomaster # OLD
>
>
> [0] - http://paste.openstack.org/show/rBCNAQJBEe9y7CKyi9aG/
> --
> Ruslanas Gžibovskis
> +370 6030 7030
>

hrm.. seems to be there:
https://hub.docker.com/r/tripleou/centos-binary-gnocchi-metricd/tags?page=1&name=current-tripleo

I see current-tripleo tags

Here are some example config and logs for your reference, although what you
have seems sane to me.

https://logserver.rdoproject.org/61/749161/1/openstack-check/tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001/f87b9d9/logs/undercloud/home/zuul/containers-prepare-parameter.yaml.txt.gz
https://logserver.rdoproject.org/61/749161/1/openstack-check/tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001/f87b9d9/logs/undercloud/var/log/tripleo-container-image-prepare-ansible.log.txt.gz
https://logserver.rdoproject.org/61/749161/1/openstack-check/tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001/f87b9d9/logs/undercloud/var/log/tripleo-container-image-prepare.log.txt.gz
https://647ea51a7c76ba5f17c8-d499b2e7288499505ef93c98963c2e35.ssl.cf5.rackcdn.com/748668/1/gate/tripleo-ci-centos-8-standalone/0c706e1/logs/undercloud/home/zuul/containers-prepare-parameters.yaml




> ___
> users mailing list
> users@lists.rdoproject.org
> http://lists.rdoproject.org/mailman/listinfo/users
>
> To unsubscribe: users-unsubscr...@lists.rdoproject.org
>
___
users mailing list
users@lists.rdoproject.org
http://lists.rdoproject.org/mailman/listinfo/users

To unsubscribe: users-unsubscr...@lists.rdoproject.org

[rdo-dev] RDO packaging for CentOS-Stream

[Wesley Hayutin]

Greetings,

Are there any public plans for building RDO packages on CentOS-Stream
available for the community to review?

Thanks 0/
___
dev mailing list
dev@lists.rdoproject.org
http://lists.rdoproject.org/mailman/listinfo/dev

To unsubscribe: dev-unsubscr...@lists.rdoproject.org

Re: [rdo-dev] RDO Cloud operations today

[Wesley Hayutin]

On Fri, Jul 10, 2020 at 1:46 AM Alan Pevec  wrote:

> > last update as of few hours ago was: rdocloud networking should be now
> > stable, uplink is not redundant, IT will work on getting back failover
> > during the day
>
> Update as of this morning:
> uplink redundancy was restored last night,
> restoring full CI pool is planned today.
>
> Cheers,
> Alan
>

Good news.. thank you
___
dev mailing list
dev@lists.rdoproject.org
http://lists.rdoproject.org/mailman/listinfo/dev

To unsubscribe: dev-unsubscr...@lists.rdoproject.org

Re: [rdo-dev] RDO Cloud operations today

[Wesley Hayutin]

On Wed, Jul 8, 2020 at 5:11 AM Alan Pevec  wrote:

> Hi all,
>
> FYI RDO Cloud is undergoing scheduled movement of some of its racks,
> control plane and infra services (www, lists, CI pool) should stay up
> all the time.
> In case of unplanned outage we'll let you know in this thread and also
> announce when those operations are finished.
> At one point there will be reduced CI pool capacity, so expect to see
> longer queues in https://review.rdoproject.org/zuul/status during the
> day.
>
> Cheers,
> Alan
>
>
Any updates on the status of the operations?


> ___
> dev mailing list
> dev@lists.rdoproject.org
> http://lists.rdoproject.org/mailman/listinfo/dev
>
> To unsubscribe: dev-unsubscr...@lists.rdoproject.org
>
>
___
dev mailing list
dev@lists.rdoproject.org
http://lists.rdoproject.org/mailman/listinfo/dev

To unsubscribe: dev-unsubscr...@lists.rdoproject.org

Re: [rdo-dev] [RDO Trunk] Renamed centos8-train builders in RDO Trunk

[Wesley Hayutin]

On Tue, Jun 16, 2020 at 11:34 AM Javier Pena  wrote:

> Dear all,
>
> A couple weeks ago, we created a new component-enabled centos8-train
> builder for RDO Trunk, using
> https://trunk.rdoproject.org/centos8-train-components as its URL.
>
> Today, we have renamed it to be the official centos8-train builder, and
> the previous non-component-enabled one has been renamed to
> centos8-train-old. To summarize:
>
> - https://trunk.rdoproject.org/centos8-train-old is the old,
> non-componentized URL
> - https://trunk.rdoproject.org/centos8-train is the new,
> component-enabled URL
>
> The TripleO CI team will work on a new component-based promotion pipeline
> for the new centos8-train builder, so we will have promoted content soon.
>
> If you were using any repo from the previous URL, and now it fails, you
> just need to add the "-old" suffix. In the future, we expect everyone to
> use the new
> component-enabled centos8-train builder, and delete the old one. Note that
> this only affects CentOS8-based builders.
>
> If you find any trouble, please do not hesitate to contact us.
>
> Regards,
> Javier
>

Awesome news Javier!
Thanks for the work and the email :)


>
> ___
> dev mailing list
> dev@lists.rdoproject.org
> http://lists.rdoproject.org/mailman/listinfo/dev
>
> To unsubscribe: dev-unsubscr...@lists.rdoproject.org
>
>
___
dev mailing list
dev@lists.rdoproject.org
http://lists.rdoproject.org/mailman/listinfo/dev

To unsubscribe: dev-unsubscr...@lists.rdoproject.org

Re: [rdo-dev] New services on rdoproject.org

[Wesley Hayutin]

On Tue, Apr 28, 2020 at 9:35 AM Daniel Pawlik  wrote:

> Hi,
>
> Today we enable new services on rdoproject.org:
> - cgit
> - codesearch
>

Great additions! Thanks Daniel!


>
> Old domain codesearch.rdoproject.org will be removed and now
> service address is: https://review.rdoproject.org/codesearch/.
> I suggest to "hard" refresh the review.rdoproject.org to
> update the site cache.
>
> If you have any problems related to this change, please do not hesitate to
> contact me.
>
>
> Regards,
> Dan
> ___
> dev mailing list
> dev@lists.rdoproject.org
> http://lists.rdoproject.org/mailman/listinfo/dev
>
> To unsubscribe: dev-unsubscr...@lists.rdoproject.org
>
___
dev mailing list
dev@lists.rdoproject.org
http://lists.rdoproject.org/mailman/listinfo/dev

To unsubscribe: dev-unsubscr...@lists.rdoproject.org

Re: [rdo-dev] Planned outage of review.rdoproject.org: 2020-03-24 13:00 UTC

[Wesley Hayutin]

On Tue, Mar 24, 2020 at 1:26 PM Tristan Cacqueray 
wrote:

>
> On Mon, Mar 16, 2020 at 14:52 Tristan Cacqueray wrote:
> > Hello folks,
> >
> > We plan to move all the control plane instances to a new cloud provided
> > by Vexxhost on 2020-03-24 13:00 UTC.
> > Major services interruption is expected during that day, including:
> > * Zuul CI not running jobs for github or opendev.
> > * RDO Trunk not building new packages.
> > * DLRN API.
> > * review.rdoproject.org and softwarefactory-project.io gerrit service.
> >
> > Let us know if you prefer a different time.
>
> The migration is now completed, the new DNS have been updated and all
> services should be working as usual.
>
> If you find any issue, please let us know and we will try to fix it as
> soon as possible.
>
> Thanks,
> -Tristan, on behalf of the Software Factory Operation Team
>

\0/  well done.
Thanks Tristan!


> ___
> dev mailing list
> dev@lists.rdoproject.org
> http://lists.rdoproject.org/mailman/listinfo/dev
>
> To unsubscribe: dev-unsubscr...@lists.rdoproject.org
>
___
dev mailing list
dev@lists.rdoproject.org
http://lists.rdoproject.org/mailman/listinfo/dev

To unsubscribe: dev-unsubscr...@lists.rdoproject.org

[rdo-dev] Fwd: [ansible-sig][kolla][openstack-ansible][osc][sdk][tripleo] OpenStack modules broken in Ansible 2.8.9

[Wesley Hayutin]

FYI..
Please make sure ansible does not get bumped to 2.8.9 we are currently at
2.8.8 in https://trunk.rdoproject.org/centos8-master/deps/latest/noarch/

Thanks!

-- Forwarded message -
From: Mark Goddard 
Date: Thu, Mar 5, 2020 at 8:28 AM
Subject: [ansible-sig][kolla][openstack-ansible][osc][sdk][tripleo]
OpenStack modules broken in Ansible 2.8.9
To: openstack-discuss 


Hi,

The 2.8.9 release of Ansible has a regression [1] which breaks the
OpenStack modules. I've proposed a simple fix, hopefully it will be
included in a 2.8.10 release soon but in the meantime you may need to
blacklist 2.8.9.

[1] https://github.com/ansible/ansible/issues/68042
[2] https://github.com/ansible/ansible/pull/68043

Cheers,
Mark
___
dev mailing list
dev@lists.rdoproject.org
http://lists.rdoproject.org/mailman/listinfo/dev

To unsubscribe: dev-unsubscr...@lists.rdoproject.org

Re: [rdo-dev] [rdo-users] [Meeting] RDO meeting (2020-02-26) minutes

[Wesley Hayutin]

On Fri, Feb 28, 2020 at 4:36 PM Wesley Hayutin  wrote:

>
>
> On Wed, Feb 26, 2020 at 7:45 AM YATIN KAREL  wrote:
>
>> ==
>> #rdo: RDO meeting - 2020-02-26
>> ==
>>
>>
>> Meeting started by ykarel at 14:02:31 UTC.  The full logs are available
>> athttp://eavesdrop.openstack.org/meetings/rdo_meeting___2020_02_26/2020/rdo_meeting___2020_02_26.2020-02-26-14.02.log.html
>> .
>>
>>
>>
>> Meeting summary
>> ---
>>
>> * roll call  (ykarel, 14:02:57)
>>
>> * CentOS 8 updates:  (ykarel, 14:06:26)
>>   * CBS buildsys-tags for ussuri on centos8 is merged  (amoralej,
>> 14:07:15)
>>   * Moving back to ansible 2.8 in CentOS8 -
>> 
>> http://lists.openstack.org/pipermail/openstack-discuss/2020-February/012820.html
>> (amoralej, 14:09:11)
>>   * LINK: https://cbs.centos.org/koji/buildinfo?buildID=28541  (ykarel,
>> 14:14:59)
>>   * LINK:
>> 
>> https://github.com/ansible/ansible/blob/stable-2.8/packaging/rpm/ansible.spec
>> (fultonj, 14:15:59)
>>   * LINK: https://cbs.centos.org/koji/buildinfo?buildID=28541  (ykarel,
>> 14:16:14)
>>   * Unpinning ussuri-uc  (amoralej, 14:20:13)
>>   * LINK:
>> https://lists.rdoproject.org/pipermail/dev/2020-February/009279.html
>> (amoralej, 14:20:21)
>>
>> Unpinning should have the following criteria.
> 1. Each centos-8 component has a component pipeline and test ( done )
> 2.  The TripleO-CI team has a centos-8 integration pipeline running and
> passing with an appropriate set of jobs ( check w/ team )
> 3.  A representative of the RDO packaging team attends the #tripleo
> meeting and updates the group on when unpinning will happen and the
> expected diff.
>
> Thanks!!
>

Please respond.. to the above criteria.
Please do NOT unpin w/o coordinating w/ the tripleo-ci team.
Thanks


>
>
>>   * once we have one more promotion, i'll propose in the ML thread to
>> unpin  (amoralej, 14:23:37)
>>   * Currently building train pkgs
>> https://review.rdoproject.org/r/#/q/topic:train-centos8  (ykarel,
>> 14:31:03)
>>
>> * chair for next week  (ykarel, 14:33:36)
>>   * ACTION: jcapitao to chair next week  (ykarel, 14:34:55)
>>
>> * open floor  (ykarel, 14:35:04)
>>
>>
>>
>> Meeting ended at 14:43:02 UTC.
>>
>>
>>
>> Action items, by person
>> ---
>>
>> * jcapitao
>>   * jcapitao to chair next week
>>
>>
>>
>> People present (lines said)
>> ---
>>
>> * amoralej (47)
>> * ykarel (45)
>> * fultonj (17)
>> * rdogerrit (10)
>> * openstack (9)
>> * jcapitao (3)
>> * jpena (3)
>> * leanderthal (1)
>> * rh-jelabarre (1)
>>
>>
>> Generated by `MeetBot`_ 0.1.4
>>
>> __
>>
>> users mailing list
>> us...@lists.rdoproject.org
>> http://lists.rdoproject.org/mailman/listinfo/users
>> To unsubscribe: users-unsubscr...@lists.rdoproject.org
>>
>>
>>
>>
>>
>>
>>
>> ___
>> dev mailing list
>> dev@lists.rdoproject.org
>> http://lists.rdoproject.org/mailman/listinfo/dev
>>
>> To unsubscribe: dev-unsubscr...@lists.rdoproject.org
>>
>
___
dev mailing list
dev@lists.rdoproject.org
http://lists.rdoproject.org/mailman/listinfo/dev

To unsubscribe: dev-unsubscr...@lists.rdoproject.org

Re: [rdo-users] [rdo-dev] [Meeting] RDO meeting (2020-02-26) minutes

[Wesley Hayutin]

On Fri, Feb 28, 2020 at 4:36 PM Wesley Hayutin  wrote:

>
>
> On Wed, Feb 26, 2020 at 7:45 AM YATIN KAREL  wrote:
>
>> ==
>> #rdo: RDO meeting - 2020-02-26
>> ==
>>
>>
>> Meeting started by ykarel at 14:02:31 UTC.  The full logs are available
>> athttp://eavesdrop.openstack.org/meetings/rdo_meeting___2020_02_26/2020/rdo_meeting___2020_02_26.2020-02-26-14.02.log.html
>> .
>>
>>
>>
>> Meeting summary
>> ---
>>
>> * roll call  (ykarel, 14:02:57)
>>
>> * CentOS 8 updates:  (ykarel, 14:06:26)
>>   * CBS buildsys-tags for ussuri on centos8 is merged  (amoralej,
>> 14:07:15)
>>   * Moving back to ansible 2.8 in CentOS8 -
>> 
>> http://lists.openstack.org/pipermail/openstack-discuss/2020-February/012820.html
>> (amoralej, 14:09:11)
>>   * LINK: https://cbs.centos.org/koji/buildinfo?buildID=28541  (ykarel,
>> 14:14:59)
>>   * LINK:
>> 
>> https://github.com/ansible/ansible/blob/stable-2.8/packaging/rpm/ansible.spec
>> (fultonj, 14:15:59)
>>   * LINK: https://cbs.centos.org/koji/buildinfo?buildID=28541  (ykarel,
>> 14:16:14)
>>   * Unpinning ussuri-uc  (amoralej, 14:20:13)
>>   * LINK:
>> https://lists.rdoproject.org/pipermail/dev/2020-February/009279.html
>> (amoralej, 14:20:21)
>>
>> Unpinning should have the following criteria.
> 1. Each centos-8 component has a component pipeline and test ( done )
> 2.  The TripleO-CI team has a centos-8 integration pipeline running and
> passing with an appropriate set of jobs ( check w/ team )
> 3.  A representative of the RDO packaging team attends the #tripleo
> meeting and updates the group on when unpinning will happen and the
> expected diff.
>
> Thanks!!
>

Please respond.. to the above criteria.
Please do NOT unpin w/o coordinating w/ the tripleo-ci team.
Thanks


>
>
>>   * once we have one more promotion, i'll propose in the ML thread to
>> unpin  (amoralej, 14:23:37)
>>   * Currently building train pkgs
>> https://review.rdoproject.org/r/#/q/topic:train-centos8  (ykarel,
>> 14:31:03)
>>
>> * chair for next week  (ykarel, 14:33:36)
>>   * ACTION: jcapitao to chair next week  (ykarel, 14:34:55)
>>
>> * open floor  (ykarel, 14:35:04)
>>
>>
>>
>> Meeting ended at 14:43:02 UTC.
>>
>>
>>
>> Action items, by person
>> ---
>>
>> * jcapitao
>>   * jcapitao to chair next week
>>
>>
>>
>> People present (lines said)
>> ---
>>
>> * amoralej (47)
>> * ykarel (45)
>> * fultonj (17)
>> * rdogerrit (10)
>> * openstack (9)
>> * jcapitao (3)
>> * jpena (3)
>> * leanderthal (1)
>> * rh-jelabarre (1)
>>
>>
>> Generated by `MeetBot`_ 0.1.4
>>
>> __
>>
>> users mailing list
>> users@lists.rdoproject.org
>> http://lists.rdoproject.org/mailman/listinfo/users
>> To unsubscribe: users-unsubscr...@lists.rdoproject.org
>>
>>
>>
>>
>>
>>
>>
>> ___
>> dev mailing list
>> d...@lists.rdoproject.org
>> http://lists.rdoproject.org/mailman/listinfo/dev
>>
>> To unsubscribe: dev-unsubscr...@lists.rdoproject.org
>>
>
___
users mailing list
users@lists.rdoproject.org
http://lists.rdoproject.org/mailman/listinfo/users

To unsubscribe: users-unsubscr...@lists.rdoproject.org

Re: [rdo-dev] [rdo-users] [Meeting] RDO meeting (2020-02-26) minutes

[Wesley Hayutin]

On Wed, Feb 26, 2020 at 7:45 AM YATIN KAREL  wrote:

> ==
> #rdo: RDO meeting - 2020-02-26
> ==
>
>
> Meeting started by ykarel at 14:02:31 UTC.  The full logs are available
> athttp://eavesdrop.openstack.org/meetings/rdo_meeting___2020_02_26/2020/rdo_meeting___2020_02_26.2020-02-26-14.02.log.html
> .
>
>
>
> Meeting summary
> ---
>
> * roll call  (ykarel, 14:02:57)
>
> * CentOS 8 updates:  (ykarel, 14:06:26)
>   * CBS buildsys-tags for ussuri on centos8 is merged  (amoralej,
> 14:07:15)
>   * Moving back to ansible 2.8 in CentOS8 -
> 
> http://lists.openstack.org/pipermail/openstack-discuss/2020-February/012820.html
> (amoralej, 14:09:11)
>   * LINK: https://cbs.centos.org/koji/buildinfo?buildID=28541  (ykarel,
> 14:14:59)
>   * LINK:
> 
> https://github.com/ansible/ansible/blob/stable-2.8/packaging/rpm/ansible.spec
> (fultonj, 14:15:59)
>   * LINK: https://cbs.centos.org/koji/buildinfo?buildID=28541  (ykarel,
> 14:16:14)
>   * Unpinning ussuri-uc  (amoralej, 14:20:13)
>   * LINK:
> https://lists.rdoproject.org/pipermail/dev/2020-February/009279.html
> (amoralej, 14:20:21)
>
> Unpinning should have the following criteria.
1. Each centos-8 component has a component pipeline and test ( done )
2.  The TripleO-CI team has a centos-8 integration pipeline running and
passing with an appropriate set of jobs ( check w/ team )
3.  A representative of the RDO packaging team attends the #tripleo meeting
and updates the group on when unpinning will happen and the expected diff.

Thanks!!


>   * once we have one more promotion, i'll propose in the ML thread to
> unpin  (amoralej, 14:23:37)
>   * Currently building train pkgs
> https://review.rdoproject.org/r/#/q/topic:train-centos8  (ykarel,
> 14:31:03)
>
> * chair for next week  (ykarel, 14:33:36)
>   * ACTION: jcapitao to chair next week  (ykarel, 14:34:55)
>
> * open floor  (ykarel, 14:35:04)
>
>
>
> Meeting ended at 14:43:02 UTC.
>
>
>
> Action items, by person
> ---
>
> * jcapitao
>   * jcapitao to chair next week
>
>
>
> People present (lines said)
> ---
>
> * amoralej (47)
> * ykarel (45)
> * fultonj (17)
> * rdogerrit (10)
> * openstack (9)
> * jcapitao (3)
> * jpena (3)
> * leanderthal (1)
> * rh-jelabarre (1)
>
>
> Generated by `MeetBot`_ 0.1.4
>
> __
>
> users mailing list
> us...@lists.rdoproject.org
> http://lists.rdoproject.org/mailman/listinfo/users
> To unsubscribe: users-unsubscr...@lists.rdoproject.org
>
>
>
>
>
>
>
> ___
> dev mailing list
> dev@lists.rdoproject.org
> http://lists.rdoproject.org/mailman/listinfo/dev
>
> To unsubscribe: dev-unsubscr...@lists.rdoproject.org
>
___
dev mailing list
dev@lists.rdoproject.org
http://lists.rdoproject.org/mailman/listinfo/dev

To unsubscribe: dev-unsubscr...@lists.rdoproject.org

Re: [rdo-users] [rdo-dev] [Meeting] RDO meeting (2020-02-26) minutes

[Wesley Hayutin]

On Wed, Feb 26, 2020 at 7:45 AM YATIN KAREL  wrote:

> ==
> #rdo: RDO meeting - 2020-02-26
> ==
>
>
> Meeting started by ykarel at 14:02:31 UTC.  The full logs are available
> athttp://eavesdrop.openstack.org/meetings/rdo_meeting___2020_02_26/2020/rdo_meeting___2020_02_26.2020-02-26-14.02.log.html
> .
>
>
>
> Meeting summary
> ---
>
> * roll call  (ykarel, 14:02:57)
>
> * CentOS 8 updates:  (ykarel, 14:06:26)
>   * CBS buildsys-tags for ussuri on centos8 is merged  (amoralej,
> 14:07:15)
>   * Moving back to ansible 2.8 in CentOS8 -
> 
> http://lists.openstack.org/pipermail/openstack-discuss/2020-February/012820.html
> (amoralej, 14:09:11)
>   * LINK: https://cbs.centos.org/koji/buildinfo?buildID=28541  (ykarel,
> 14:14:59)
>   * LINK:
> 
> https://github.com/ansible/ansible/blob/stable-2.8/packaging/rpm/ansible.spec
> (fultonj, 14:15:59)
>   * LINK: https://cbs.centos.org/koji/buildinfo?buildID=28541  (ykarel,
> 14:16:14)
>   * Unpinning ussuri-uc  (amoralej, 14:20:13)
>   * LINK:
> https://lists.rdoproject.org/pipermail/dev/2020-February/009279.html
> (amoralej, 14:20:21)
>
> Unpinning should have the following criteria.
1. Each centos-8 component has a component pipeline and test ( done )
2.  The TripleO-CI team has a centos-8 integration pipeline running and
passing with an appropriate set of jobs ( check w/ team )
3.  A representative of the RDO packaging team attends the #tripleo meeting
and updates the group on when unpinning will happen and the expected diff.

Thanks!!


>   * once we have one more promotion, i'll propose in the ML thread to
> unpin  (amoralej, 14:23:37)
>   * Currently building train pkgs
> https://review.rdoproject.org/r/#/q/topic:train-centos8  (ykarel,
> 14:31:03)
>
> * chair for next week  (ykarel, 14:33:36)
>   * ACTION: jcapitao to chair next week  (ykarel, 14:34:55)
>
> * open floor  (ykarel, 14:35:04)
>
>
>
> Meeting ended at 14:43:02 UTC.
>
>
>
> Action items, by person
> ---
>
> * jcapitao
>   * jcapitao to chair next week
>
>
>
> People present (lines said)
> ---
>
> * amoralej (47)
> * ykarel (45)
> * fultonj (17)
> * rdogerrit (10)
> * openstack (9)
> * jcapitao (3)
> * jpena (3)
> * leanderthal (1)
> * rh-jelabarre (1)
>
>
> Generated by `MeetBot`_ 0.1.4
>
> __
>
> users mailing list
> users@lists.rdoproject.org
> http://lists.rdoproject.org/mailman/listinfo/users
> To unsubscribe: users-unsubscr...@lists.rdoproject.org
>
>
>
>
>
>
>
> ___
> dev mailing list
> d...@lists.rdoproject.org
> http://lists.rdoproject.org/mailman/listinfo/dev
>
> To unsubscribe: dev-unsubscr...@lists.rdoproject.org
>
___
users mailing list
users@lists.rdoproject.org
http://lists.rdoproject.org/mailman/listinfo/users

To unsubscribe: users-unsubscr...@lists.rdoproject.org

Re: [rdo-dev] [tripleo] missing centos-8 rpms for kolla builds

[Wesley Hayutin]

On Mon, Feb 24, 2020 at 8:55 AM Mark Goddard  wrote:

> On Wed, 29 Jan 2020 at 11:31, Alfredo Moralejo Alonso
>  wrote:
> >
> >
> >
> > On Tue, Jan 28, 2020 at 5:53 PM Mark Goddard  wrote:
> >>
> >> On Tue, 28 Jan 2020 at 15:18, Mark Goddard  wrote:
> >> >
> >> > On Mon, 27 Jan 2020 at 09:18, Radosław Piliszek
> >> >  wrote:
> >> > >
> >> > > I know it was for masakari.
> >> > > Gaëtan had to grab crmsh from opensuse:
> >> > >
> http://download.opensuse.org/repositories/network:/ha-clustering:/Stable/CentOS_CentOS-7/
> >> > >
> >> > > -yoctozepto
> >> >
> >> > Thanks Wes for getting this discussion going. I've been looking at
> >> > CentOS 8 today and trying to assess where we are. I created an
> >> > Etherpad to track status:
> >> > https://etherpad.openstack.org/p/kolla-centos8
> >>
> >
> > uwsgi and etcd are now available in rdo dependencies repo. Let me know
> if you find some issue with it.
>
> I've been working on the backport of kolla CentOS 8 patches to the
> stable/train branch. It looks like these packages which were added to
> master are not present in Train.
>
>
I'll help check in on that for you Mark.
Thank you!!



> >
> >>
> >> We are seeing an odd DNF error sometimes. DNF exits 141 with no error
> >> code when installing packages. It often happens on the rabbitmq and
> >> grafana images. There is a prompt about importing GPG keys prior to
> >> the error.
> >>
> >> Example:
> https://4eff4bb69c321960be39-770d619687de1bce0976465c40e4e9ca.ssl.cf2.rackcdn.com/693544/33/check/kolla-ansible-centos8-source-mariadb/93a8351/primary/logs/build/000_FAILED_kolla-toolbox.log
> >>
> >> Related bug report? https://github.com/containers/libpod/issues/4431
> >>
> >> Anyone familiar with it?
> >>
> >
> > Didn't know about this issue.
> >
> > BTW, there is rabbitmq-server in RDO dependencies repo if you are
> interested in using it from there instead of rabbit repo.
> >
> >> >
> >> > >
> >> > > pon., 27 sty 2020 o 10:13 Marcin Juszkiewicz
> >> > >  napisał(a):
> >> > > >
> >> > > > W dniu 27.01.2020 o 09:48, Alfredo Moralejo Alonso pisze:
> >> > > > > How is crmsh used in these images?, ha packages included in
> >> > > > > HighAvailability repo in CentOS includes pcs and some crm_*
> commands in pcs
> >> > > > > and pacemaker-cli packages. IMO, tt'd be good to switch to
> those commands
> >> > > > > to manage the cluster.
> >> > > >
> >> > > > No idea. Gaëtan Trellu may know - he created those images.
> >> > > >
> >> > >
> >>
>
>
___
dev mailing list
dev@lists.rdoproject.org
http://lists.rdoproject.org/mailman/listinfo/dev

To unsubscribe: dev-unsubscr...@lists.rdoproject.org

Re: [rdo-dev] Status of RDO Trunk Ussuri on CentOS 7 and transition to CentOS 8

[Wesley Hayutin]

On Thu, Feb 20, 2020 at 11:56 AM Mike Burns  wrote:

> So, if I understand this right...
>
> * We're keeping the centos7 and centos8 versions of Ussuri consistent
> * There is no actual build of the latest commits built on either centos7
> or centos8
> * There is no possible way to have an Ussuri release on Centos7
>
> Assuming the above is true, I would say to turn off all the centos7 stuff
> and focus all effort on getting Ussuri latest built on centos8.  There does
> not appear to be any value in continuing work on centos7 if we can't
> actually deliver it at the end of the day.
>
> Mike
>

Agreeing and building on what Mike said.. this certainly raises the
priority on CentOS-8.  There are still tripleo patches incoming that will
not be pinned ( I think )
Having a better understanding here of what has been pinned and what is in
progress w/o consulting rdo-info all the time would be handy.

I think the check, gate and promotion jobs for centos-7 ussuri need to stay
in place, however they should just always pass which would lower the amount
of work w/ regards to centos-7 but not eliminate it.   It could also be the
case that tripleo breaks itself from time to time, but I'm sure Emilien
would never let that happen as he's a robot and not human.

I digress


>
> On Thu, Feb 20, 2020 at 9:05 AM Alfredo Moralejo Alonso <
> amora...@redhat.com> wrote:
>
>>
>> Hi,
>>
>> I'd like to open a discussion about the status of RDO Ussuri
>> repositories on CentOS7.
>>
>> As you know RDO and upstream teams (kolla, puppet-openstack, TripleO,
>> TripleO CI, etc...) have been working to switch to CentOS8 during last
>> few weeks.
>>
>> In order to make the transition easier from CentOS 7 to CentOS 8, RDO is
>> still maintaining Trunk repos consistent for both CentOS 7/Python 2 and
>> CentOS 8/Python 3. As OpenStack projects have been dropping support for P
>> ython 2, we've started pinning them to the last commit working with P
>> ython 2[1], we were expecting that transition will finish soon but it's
>> still going on. Over time, the number of pinned packages has been
>> growing including services and Oslo libraries where we can't follow
>> upper-constraints anymore[2]. Recently, Kolla has removed support for
>> CentOS 7 so i doubt it makes sense to keep pinning packages to keep RDO
>> Trunk consistent artificially and continue running promotion pipelines on a
>> repo with so many outdated packages. Also, pinning these projects makes
>> that changes needed for CentOS 8 will not be in RDO and would need to be
>> backported manually to each package. My proposal is:
>>
>> - Unpin all packages in Ussuri to follow master trunk, or versions in u
>> pper-constraints (for clients and libraries).
>> - RDO Ussuri on CentOS 7 repo consistent link will not move anymore (so
>> no more promotions based on it).
>> - We will keep running centos7-master DLRN builder, so that packages
>> still builing with Python 2 will be available in current repo [3] to be
>> used by teams needing them until migration to CentOS 8 is finished
>> everywhere.
>> - Projects which already have CentOS 8 jobs gating in master branch can
>> remove CentOS 7 ones.
>>
>> We understand this can add some pressure on moving to CentOS8 to the
>> teams working on it, but I'd say it's already a priority and it's justified
>> at this stage.
>>
>> What do you think about this plan?, is there any reason to keep CentOS 7
>> artificially consistent and promoting at this point of the transition to
>> CentOS 8?
>>
>> Best regards,
>>
>> Alfredo
>>
>> [1] https://review.rdoproject.org/r/#/q/topic:pin-py2
>> [2] https://review.rdoproject.org/r/#/c/24796/
>> [3] http://trunk.rdoproject.org/centos7-master/current
>> ___
>> dev mailing list
>> dev@lists.rdoproject.org
>> http://lists.rdoproject.org/mailman/listinfo/dev
>>
>> To unsubscribe: dev-unsubscr...@lists.rdoproject.org
>>
> ___
> dev mailing list
> dev@lists.rdoproject.org
> http://lists.rdoproject.org/mailman/listinfo/dev
>
> To unsubscribe: dev-unsubscr...@lists.rdoproject.org
>
___
dev mailing list
dev@lists.rdoproject.org
http://lists.rdoproject.org/mailman/listinfo/dev

To unsubscribe: dev-unsubscr...@lists.rdoproject.org

Re: [rdo-users] [rdo-dev] Status of RDO Trunk Ussuri on CentOS 7 and transition to CentOS 8

[Wesley Hayutin]

On Thu, Feb 20, 2020 at 11:56 AM Mike Burns  wrote:

> So, if I understand this right...
>
> * We're keeping the centos7 and centos8 versions of Ussuri consistent
> * There is no actual build of the latest commits built on either centos7
> or centos8
> * There is no possible way to have an Ussuri release on Centos7
>
> Assuming the above is true, I would say to turn off all the centos7 stuff
> and focus all effort on getting Ussuri latest built on centos8.  There does
> not appear to be any value in continuing work on centos7 if we can't
> actually deliver it at the end of the day.
>
> Mike
>

Agreeing and building on what Mike said.. this certainly raises the
priority on CentOS-8.  There are still tripleo patches incoming that will
not be pinned ( I think )
Having a better understanding here of what has been pinned and what is in
progress w/o consulting rdo-info all the time would be handy.

I think the check, gate and promotion jobs for centos-7 ussuri need to stay
in place, however they should just always pass which would lower the amount
of work w/ regards to centos-7 but not eliminate it.   It could also be the
case that tripleo breaks itself from time to time, but I'm sure Emilien
would never let that happen as he's a robot and not human.

I digress


>
> On Thu, Feb 20, 2020 at 9:05 AM Alfredo Moralejo Alonso <
> amora...@redhat.com> wrote:
>
>>
>> Hi,
>>
>> I'd like to open a discussion about the status of RDO Ussuri
>> repositories on CentOS7.
>>
>> As you know RDO and upstream teams (kolla, puppet-openstack, TripleO,
>> TripleO CI, etc...) have been working to switch to CentOS8 during last
>> few weeks.
>>
>> In order to make the transition easier from CentOS 7 to CentOS 8, RDO is
>> still maintaining Trunk repos consistent for both CentOS 7/Python 2 and
>> CentOS 8/Python 3. As OpenStack projects have been dropping support for P
>> ython 2, we've started pinning them to the last commit working with P
>> ython 2[1], we were expecting that transition will finish soon but it's
>> still going on. Over time, the number of pinned packages has been
>> growing including services and Oslo libraries where we can't follow
>> upper-constraints anymore[2]. Recently, Kolla has removed support for
>> CentOS 7 so i doubt it makes sense to keep pinning packages to keep RDO
>> Trunk consistent artificially and continue running promotion pipelines on a
>> repo with so many outdated packages. Also, pinning these projects makes
>> that changes needed for CentOS 8 will not be in RDO and would need to be
>> backported manually to each package. My proposal is:
>>
>> - Unpin all packages in Ussuri to follow master trunk, or versions in u
>> pper-constraints (for clients and libraries).
>> - RDO Ussuri on CentOS 7 repo consistent link will not move anymore (so
>> no more promotions based on it).
>> - We will keep running centos7-master DLRN builder, so that packages
>> still builing with Python 2 will be available in current repo [3] to be
>> used by teams needing them until migration to CentOS 8 is finished
>> everywhere.
>> - Projects which already have CentOS 8 jobs gating in master branch can
>> remove CentOS 7 ones.
>>
>> We understand this can add some pressure on moving to CentOS8 to the
>> teams working on it, but I'd say it's already a priority and it's justified
>> at this stage.
>>
>> What do you think about this plan?, is there any reason to keep CentOS 7
>> artificially consistent and promoting at this point of the transition to
>> CentOS 8?
>>
>> Best regards,
>>
>> Alfredo
>>
>> [1] https://review.rdoproject.org/r/#/q/topic:pin-py2
>> [2] https://review.rdoproject.org/r/#/c/24796/
>> [3] http://trunk.rdoproject.org/centos7-master/current
>> ___
>> dev mailing list
>> d...@lists.rdoproject.org
>> http://lists.rdoproject.org/mailman/listinfo/dev
>>
>> To unsubscribe: dev-unsubscr...@lists.rdoproject.org
>>
> ___
> dev mailing list
> d...@lists.rdoproject.org
> http://lists.rdoproject.org/mailman/listinfo/dev
>
> To unsubscribe: dev-unsubscr...@lists.rdoproject.org
>
___
users mailing list
users@lists.rdoproject.org
http://lists.rdoproject.org/mailman/listinfo/users

To unsubscribe: users-unsubscr...@lists.rdoproject.org

[rdo-dev] [tripleo] missing centos-8 rpms for kolla builds

[Wesley Hayutin]

Greetings,

I know the ceph repo is in progress.
TripleO / RDO is not releasing opendaylight

Can the RDO team comment on the rest of the missing packages here please?

Thank you!!

https://review.opendev.org/#/c/699414/9/kolla/image/build.py

 NOTE(mgoddard): Mark images with missing dependencies as unbuildable for
# CentOS 8.
'centos8': {
"barbican-api",  # Missing uwsgi-plugin-python3
"ceph-base", # Missing Ceph repo
"cinder-base",   # Missing Ceph repo
"collectd",  # Missing collectd-ping and
 # collectd-sensubility packages
"elasticsearch", # Missing elasticsearch repo
"etcd",  # Missing etcd package
"fluentd",   # Missing td-agent repo
"glance-base",   # Missing Ceph repo
"gnocchi-base",  # Missing Ceph repo
"hacluster-base",# Missing hacluster repo
"ironic-conductor",  # Missing shellinabox package
"kibana",# Missing elasticsearch repo
"manila-share",  # Missing Ceph repo
"mongodb",   # Missing mongodb and mongodb-server
packages
"monasca-grafana",   # Using python2
"nova-compute",  # Missing Ceph repo
"nova-libvirt",  # Missing Ceph repo
"nova-spicehtml5proxy",  # Missing spicehtml5 package
"opendaylight",  # Missing opendaylight repo
"ovsdpdk",   # Not supported on CentOS
"sensu-base",# Missing sensu package
"tgtd",  # Not supported on CentOS 8
},

'centos8+source': {
"barbican-base", # Missing uwsgi-plugin-python3
"bifrost-base",  # Bifrost does not support CentOS 8
"cyborg-agent",  # opae-sdk does not support CentOS 8
"freezer-base",  # Missing package trickle
"masakari-monitors", # Missing hacluster repo
"zun-compute",   # Missing Ceph repo
___
dev mailing list
dev@lists.rdoproject.org
http://lists.rdoproject.org/mailman/listinfo/dev

To unsubscribe: dev-unsubscr...@lists.rdoproject.org

Re: [rdo-dev] Proposing Sorin Sbarnea (zbr) and Marios Andreou (marios) as core on review.rdoproject.org repos config and rdo-jobs

[Wesley Hayutin]

+1


On Fri, Aug 23, 2019 at 8:31 AM Ronelle Landy  wrote:

> +1
> Both would be great additions
>
> On Fri, Aug 23, 2019 at 10:22 AM Chandan kumar 
> wrote:
>
>> Hello,
>>
>> I'd like to propose Sorin and Marios as new cores on the
>> review.rdoproject.org repos config and rdo-jobs.
>>
>> Sorin and Marios have done amazing work on these repos by providing
>> valuable feedback and knowledge on the reviews and making sure it can
>> be tested via bringing molecule and avoiding repetition.
>>
>> They are both knowledgeable in the design and working of TripleO jobs and
>> done
>> an amazing work while bringing RHEL8 on RDO.
>>
>> Having them as a core on these repos would be a great addition.
>>
>> The vote will end on Friday, Aug 30.
>>
>> Thanks,
>>
>> Chandan Kumar
>> ___
>> dev mailing list
>> dev@lists.rdoproject.org
>> http://lists.rdoproject.org/mailman/listinfo/dev
>>
>> To unsubscribe: dev-unsubscr...@lists.rdoproject.org
>>
> ___
> dev mailing list
> dev@lists.rdoproject.org
> http://lists.rdoproject.org/mailman/listinfo/dev
>
> To unsubscribe: dev-unsubscr...@lists.rdoproject.org
>
___
dev mailing list
dev@lists.rdoproject.org
http://lists.rdoproject.org/mailman/listinfo/dev

To unsubscribe: dev-unsubscr...@lists.rdoproject.org

Re: [rdo-dev] [infra][tripleo-ci] Disk space usage in logs.rdoproject.org

[Wesley Hayutin]

On Thu, Jun 13, 2019 at 8:55 AM Javier Pena  wrote:

>
>
> --
>
>
>
> On Thu, Jun 13, 2019 at 8:22 AM Javier Pena  wrote:
>
>> Hi all,
>>
>> For the last few days, I have been monitoring a spike in disk space
>> utilization for logs.rdoproject.org. The current situation is:
>>
>> - 94% of space used, with less than 140GB out of 2TB available.
>> - The log pruning script has been reclaiming less space than we are using
>> for new logs during this week.
>> - I expect the situation to improve over the weekend, but we're
>> definitely running out of space.
>>
>> I have looked at a random job (https://review.opendev.org/639324, patch
>> set 26), and found that each run is consuming 1.2 GB of disk space in logs.
>> The worst offenders I have found are:
>>
>> - atop.bin.gz files (one per job, 8 jobs per recheck), ranging between 15
>> and 40 MB each
>> - logs/undercloud/home/zuul/tempest/.stackviz directory on
>> tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001 jobs, which is a
>> virtualenv eating up 81 MB.
>>
>
> Can we sync up w/ how you are calculating these results as they do not
> match our results.
> I see each job consuming about 215M of space, we are close on stackviz
> being 83M. Oddly I don't see atop.bin.gz in our calculations so I'll have
> to look into that.
>
> I've checked it directly using du on the logserver. By 1.2 GB I meant the
> aggregate of the 8 jobs running for a single patchset. PS26 is currently
> using 2.5 GB and had one recheck.
>
> About the atop.bin.gz file:
>
> # find . -name atop.bin.gz -exec du -sh {} \;
> 16M
>  
> ./tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-queens-branch/042cb8f/logs/undercloud/var/log/atop.bin.gz
> 16M
>  
> ./tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-queens-branch/e4171d7/logs/undercloud/var/log/atop.bin.gz
> 28M
>  
> ./tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-rocky-branch/ffd4de9/logs/undercloud/var/log/atop.bin.gz
> 26M
>  
> ./tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-rocky-branch/34d44bf/logs/undercloud/var/log/atop.bin.gz
> 25M
>  
> ./tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-stein-branch/b89761d/logs/undercloud/var/log/atop.bin.gz
> 24M
>  
> ./tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-stein-branch/9ade834/logs/undercloud/var/log/atop.bin.gz
> 29M
>  
> ./tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset053/a10447d/logs/undercloud/var/log/atop.bin.gz
> 44M
>  
> ./tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset053/99a5f9f/logs/undercloud/var/log/atop.bin.gz
> 15M
>  
> ./tripleo-ci-centos-7-multinode-1ctlr-featureset010/c8a8c60/logs/subnode-2/var/log/atop.bin.gz
> 33M
>  
> ./tripleo-ci-centos-7-multinode-1ctlr-featureset010/c8a8c60/logs/undercloud/var/log/atop.bin.gz
> 16M
>  
> ./tripleo-ci-centos-7-multinode-1ctlr-featureset010/73ef532/logs/subnode-2/var/log/atop.bin.gz
> 33M
>  
> ./tripleo-ci-centos-7-multinode-1ctlr-featureset010/73ef532/logs/undercloud/var/log/atop.bin.gz
> 40M
>  
> ./tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035/109d5ae/logs/undercloud/var/log/atop.bin.gz
> 45M
>  
> ./tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035/c2ebeae/logs/undercloud/var/log/atop.bin.gz
> 39M
>  
> ./tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001/7fe5bbb/logs/undercloud/var/log/atop.bin.gz
> 16M
>  
> ./tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001/5e6cb0f/logs/undercloud/var/log/atop.bin.gz
> 40M
>  
> ./tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039/c6bf5ea/logs/undercloud/var/log/atop.bin.gz
> 40M
>  
> ./tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039/6ec5ac6/logs/undercloud/var/log/atop.bin.gz
>
> Can I safely delete all .stackviz directories? I guess that would give us
> some breathing room.
>

Yup, go for it


>
> Regards,
> Javier
>
> Each job reports the size of the logs e.g. [1]
>
> http://logs.rdoproject.org/24/639324/26/openstack-check/tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-stein-branch/9ade834/logs/quickstart_files/log-size.txt
>
>
>> As a temporary measure, I am reducing log retention from 21 days to 14,
>> but we still need to reduce the rate at which we are uploading logs. Would
>> it be possible to check the oooq-generated logs and see where we can
>> reduce? These jobs are by far the ones consuming most space.
>>
>> Thanks,
>> Javier
>> ___
>> dev mailing list
>> dev@lists.rdoproject.org
>> http://lists.rdoproject.org/mailman/listinfo/dev
>>
>> To unsubscribe: dev-unsubscr...@lists.rdoproject.org
>>
>
>
___
dev mailing list
dev@lists.rdoproject.org
http://lists.rdoproject.org/mailman/listinfo/dev

To unsubscribe: dev-unsubscr...@lists.rdoproject.org

Re: [rdo-dev] [infra][tripleo-ci] Disk space usage in logs.rdoproject.org

[Wesley Hayutin]

On Thu, Jun 13, 2019 at 8:22 AM Javier Pena  wrote:

> Hi all,
>
> For the last few days, I have been monitoring a spike in disk space
> utilization for logs.rdoproject.org. The current situation is:
>
> - 94% of space used, with less than 140GB out of 2TB available.
> - The log pruning script has been reclaiming less space than we are using
> for new logs during this week.
> - I expect the situation to improve over the weekend, but we're definitely
> running out of space.
>
> I have looked at a random job (https://review.opendev.org/639324, patch
> set 26), and found that each run is consuming 1.2 GB of disk space in logs.
> The worst offenders I have found are:
>
> - atop.bin.gz files (one per job, 8 jobs per recheck), ranging between 15
> and 40 MB each
> - logs/undercloud/home/zuul/tempest/.stackviz directory on
> tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001 jobs, which is a
> virtualenv eating up 81 MB.
>

Can we sync up w/ how you are calculating these results as they do not
match our results.
I see each job consuming about 215M of space, we are close on stackviz
being 83M. Oddly I don't see atop.bin.gz in our calculations so I'll have
to look into that.

Each job reports the size of the logs e.g. [1]
http://logs.rdoproject.org/24/639324/26/openstack-check/tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-stein-branch/9ade834/logs/quickstart_files/log-size.txt



>
> As a temporary measure, I am reducing log retention from 21 days to 14,
> but we still need to reduce the rate at which we are uploading logs. Would
> it be possible to check the oooq-generated logs and see where we can
> reduce? These jobs are by far the ones consuming most space.
>
> Thanks,
> Javier
> ___
> dev mailing list
> dev@lists.rdoproject.org
> http://lists.rdoproject.org/mailman/listinfo/dev
>
> To unsubscribe: dev-unsubscr...@lists.rdoproject.org
>
___
dev mailing list
dev@lists.rdoproject.org
http://lists.rdoproject.org/mailman/listinfo/dev

To unsubscribe: dev-unsubscr...@lists.rdoproject.org

Re: [OpenStack-Infra] Low Key Monday Evening Get Together

[Wesley Hayutin]

Just some local input..
That is a long way to go from downtown.  There are plenty of outdoor venues
with great beer a little closer. You might want to try a personal favorite
[1], there are ton of other breweries around odell as well.  Hard to go
wrong there.  Either way, enjoy your time in Denver!

[1] https://www.odellbrewing.com/taproom-denver/

On Wed, Apr 24, 2019 at 3:43 PM Clark Boylan  wrote:

> On Wed, Apr 24, 2019, at 9:14 AM, Clark Boylan wrote:
> > Hello Infra!
> >
> > Monday evening is looking like a good night to have a low key informal
> > team gathering. The only official event on the calendar is the
> > Marketplace Mixer which runs until 7pm. Weather permitting I'd like to
> > head back to Lowry Beer Garden (where we've gone at past PTGs). It is a
> > bit out of the way from the conference center so we will need to
> > coordinate uber/lyft transport but that hasn't been a problem in the
> > past.
> >
> > Lets meet at 6:30pm (I can send specific location once onsite) and head
> > over to Lowry. If the weather looks terrible I can call ahead and see
> > if their indoor area is open and if they say it is "meh" we'll find
> > something closer to the conference center. Also, they close at 9pm
> > which should force us to get some sleep :)
> >
> > Finally, Monday is a good day because it is gertty's birthday. Hope to
> > see you then.
>
> Hey, it was pointed out that there is a really good chance that Zuul will
> be confirmed as a top level OSF project at the board meeting on Sunday.
> This seems like an excellent thing to celebrate as well, and given the
> overlap between Zuul and Infra teams I think we should celebrate that
> together.
>
> Don’t want to put the cart before the horse though, so we can wait until
> we know for sure before we put a label on it. But please go ahead and mark
> Monday evening for Zuul too just in case.
>
> Also looking at the weather more closely it will be about 50F/11C with
> rain and wind Monday evening, so Lowry may not be the best choice. It has
> been pointed out that For The Win Denver would be willing to take us as an
> informal group and we can play arcade games indoors instead. They have food
> and drinks too.
>
> As things firm up I'll send an update to this email with concrete time and
> location details.
>
> Clark
>
> ___
> OpenStack-Infra mailing list
> OpenStack-Infra@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
___
OpenStack-Infra mailing list
OpenStack-Infra@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra

Re: [rdo-dev] [infra][tripleoci] ppc64le container images in registry.rdoproject.org

[Wesley Hayutin]

rebooting this conversation...  see inline at the bottom

On Mon, Mar 25, 2019 at 11:29 AM Javier Pena  wrote:

>
>
> --
>
> On Fri, Mar 22, 2019 at 8:35 PM Javier Pena  wrote:
>
>>
>>
>> - Original Message -
>> > I've been working with mjturek and baha on this a bit.  I've responded
>> inline
>> > below, but also want to clarify on the desired workflow.
>> >
>> > TL;DR: The desired workflow is to have ppc64le and x86_64 seamlessly
>> > integrated and uploaded.  This can be done with docker manifest list
>> images.
>> >
>> > The following link explains in greater detail:
>> > https://docs.docker.com/registry/spec/manifest-v2-2/
>> >
>> > The process boils down to the following steps:
>> >
>> > 1) Upload an image of the first architecture  (ex:
>> image1:x86_64_01012019)
>> > 2) Upload an image of the second architecture  (ex:
>> image1:ppc64le_01012019)
>> > 3) Upload manifest list image of the image  (ex: image1:01012019)
>> >
>>
>> This is one of the details where I had my doubts. Currently, the images
>> uploaded to the registry use the following naming convention:
>>
>>
>> tripleomaster/centos-binary-neutron-l3-agent:42a882962919b867c91a182b83acca6d8004096e_ee467b40
>>
>>
>> Where:
>>
>> - tripleomaster is associated to the release (we have tripleomaster,
>> tripleostein, tripleorocky...)
>> - centos is associated to the OS (we have centos and fedora)
>> - 42a882962919b867c91a182b83acca6d8004096e_ee467b40 refers to the
>> repository in trunk.rdoproject.org used to build the image (commit hash
>> and short distro hash)
>>
>> If we want to go multi-arch, we need to change that tag to include the
>> architecture, is this correct? Otherwise, we could have conflicts between
>> the x86_64 and ppc64le pipelines trying to upload the same image.
>>
>
> Yup.  The idea is that the enpoint URL
> (tripleomaster/centos-binary-neutron-l3-agent:42a882962919b867c91a182b83acca6d8004096e_ee467b40)
> is a container manifest.  Where we include the arch would be with an
> additional tag:
>
> tripleomaster/centos-binary-neutron-l3-agent:42a882962919b867c91a182b83acca6d8004096e_ee467b40_$arch
> but nothing else should change and *explicitly* do not want different orgs
> per architecture.  So the publish pipeline would look like:
>
>
>- Each architecture builds and publishes all the containers per branch
>and OS [1] all the containers and publishes a container image/layer to:
>
>'tripleo%(branch)s/%(os)s-%(build_type)s-%(container)s:%(repo)s_%(arch)s'
>- Then checks to see if the manifest exists.
>manifest =
>'tripleo%(branch)s/%(os)s-%(build_type)s-%(container)s:%(repo)s'
>if exists(manifest):
>
>
> add_to_manifest(arch_layer=tripleo%(branch)s/%(os)s-%(build_type)s-%(container)s:%(repo)s_%(arch)s')
>else:
>
>
> create_manifest(arch_layer=tripleo%(branch)s/%(os)s-%(build_type)s-%(container)s:%(repo)s_%(arch)s')
>
> I have been running some tests to check how this could work, and I've
> found an issue. It looks like the OpenShift Registry (what we use for our
> RDO Registry) does not properly support manifest lists, see [2]. It
> actually failed for me when I tried it, while a plain Docker registry
> worked (using manifest-tool).
>
> Would the manifest upload be required in the RDO Registry (which is used
> as an intermediate step), or just in DockerHub (which is used for actual
> content delivery)? If it's the second case, we're still fine.
>
> Regards,
> Javier
>
> [2] -
> https://trello.com/c/4EcAIJrd/1303-5-quayregistry-add-support-for-manifest-list-rd
>
> This shouldn't break existing consumers as docker and podman both do the
> correct thing when encountering a manifest.  and does mean that multi-arch
> consumers can use the same URL scheme.  This is how downstream currently
> works
>
> It's possible and possibly even desirable, due to resource constraints,
> for the ppc64le build to be triggered only when updating
> current-passed-ci.  That's exactly what we discussed in Dublin.
>
> Tony.
>
> [1] for ppc64le we're starting with centos and master but over time this
> would need to grow out from master to include stein, u etc etc  We haven't
> looked at Fedora due to using centos CI but if Fedora is going to stick
> around we can work on that too.
>
>
> ___
> dev mailing list
> dev@lists.rdoproject.org
> http://lists.rdoproject.org/mailman/listinfo/dev
>
> To unsubscribe: dev-unsubscr...@lists.rdoproject.org



OK... so here we are.
Trevor, myself and Steve hashed out a PPC64LE workflow that works alongside
the x86_64 workflow.
Our notes are here [1] .

Although x86_64 and PPC64LE should try their best to use the same container
build methods and process the workflows are completely independent and can
remain that way.   The convergence occurs on the tripleo-ci promotion
server after reading test results from the dlrn_api ( notes in the etherpad
)

Some additional points.
* Alan Pevec and I both agree gett

Re: [rdo-dev] [infra][tripleoci] ppc64le container images in registry.rdoproject.org

[Wesley Hayutin]

On Fri, Mar 22, 2019 at 3:36 AM Javier Pena  wrote:

>
>
> - Original Message -
> > I've been working with mjturek and baha on this a bit.  I've responded
> inline
> > below, but also want to clarify on the desired workflow.
> >
> > TL;DR: The desired workflow is to have ppc64le and x86_64 seamlessly
> > integrated and uploaded.  This can be done with docker manifest list
> images.
> >
> > The following link explains in greater detail:
> > https://docs.docker.com/registry/spec/manifest-v2-2/
> >
> > The process boils down to the following steps:
> >
> > 1) Upload an image of the first architecture  (ex:
> image1:x86_64_01012019)
> > 2) Upload an image of the second architecture  (ex:
> image1:ppc64le_01012019)
> > 3) Upload manifest list image of the image  (ex: image1:01012019)
> >
>
> This is one of the details where I had my doubts. Currently, the images
> uploaded to the registry use the following naming convention:
>
>
> tripleomaster/centos-binary-neutron-l3-agent:42a882962919b867c91a182b83acca6d8004096e_ee467b40
>
>
> Where:
>
> - tripleomaster is associated to the release (we have tripleomaster,
> tripleostein, tripleorocky...)
> - centos is associated to the OS (we have centos and fedora)
> - 42a882962919b867c91a182b83acca6d8004096e_ee467b40 refers to the
> repository in trunk.rdoproject.org used to build the image (commit hash
> and short distro hash)
>
> If we want to go multi-arch, we need to change that tag to include the
> architecture, is this correct? Otherwise, we could have conflicts between
> the x86_64 and ppc64le pipelines trying to upload the same image.
>
> Regards,
> Javier
>

Right.. I'm agreeing w/ you..

I'm recommending the following change
tripleomaster -> tripleomaster_x86_64
and add
tripleomaster_ppc64le


>
>
> > Step 3 is essentially just pushing a JSON body that has descriptors and
> > references to the other two images, such that when someone does a pull
> > request of the manifest list image, it will gather the appropriate
> > architecture for that image based on the host's architecture.
> >
> > -Trevor
> >
> > PS. If I've missed something important with the overall concerns here I
> > apologize, but thought it necessary to spell out the goal as I understand
> > it.
> >
> > > On Mar 21, 2019, at 12:28 PM, Javier Pena  wrote:
> > >
> > >
> > > - Original Message -
> > >> Hi all,
> > >>
> > >> Over the last few weeks, mjturek and baha have been busy working on a
> set
> > >> of
> > >> periodic jobs to build TripleO images for the ppc64le arch [1].
> > >>
> > >> The current missing step is publishing those images, and they are
> > >> proposing
> > >> to push those to the RDO Registry instance at registry.rdoproject.org
> ,
> > >> just
> > >> like we do with our TripleO images. I have tried to understand the
> > >> requirements, and would like to get input on the following topics:
> > >>
> > >> - Which namespace would these images use? Based on some logs [2] it
> looks
> > >> like they use tripleomaster-ppc64le, will they also push the images to
> > >> that
> > >> namespace?
> >
> > I have no experience in namespaces inside of a registry or how that
> > differentiates images from one another, but the images should be pushed
> (in
> > my opinion) to the same location in which the x86 images reside.
> >
> > >> - Could this create any conflicts with our current promotion pipeline?
> >
> > This should not cause conflicts in current promotion pipeline, as the
> process
> > should be an extension to existing functionality.
> >
> > >> - Is registry.rdo the right place for those images? I'm not familiar
> with
> > >> the
> > >> next steps for ppc64le images after that (will it then go through a
> > >> promotion pipeline?), so that might affect.
> >
> > If the x86 images exist in registry.rdo, then the ppc64le (and any other
> > architecture image) should exist there as well.  I can't think of a
> reason
> > to differentiate between architectures when the desired result is parity
> and
> > support of more architectures.
> >
> > >>
> > >> If we decide to upload the images to images.rdo, we'll need to do the
> > >
> > > Correction: this should read "registry.rdo"
> > >
> > >> following:
> > >>
> > >> - Create the tripleomaster-ppc64le namespace in registry.rdo,
> following a
> > >> similar pattern to [3].
> > >> - Schedule a short registry downtime to increase its disk space,
> since it
> > >> is
> > >> currently near its limit.
> >
> > This is definitely necessary, given the capacity requirement will double,
> > give or take, to support the additional architecture.
> >
> > >> - Update the job at ci.centos to include the REGISTRY_PASSWORD
> environment
> > >> variable with the right token (see [4]). This is missing today, and
> > >> causing
> > >> the job failure.
> > >>
> > >> Once we get input from all interested parties, we will decide on the
> next
> > >> steps.
> > >>
> > >> Thanks,
> > >> Javier
> > >>
> > >>
> > >> [1] -
> > >>
> https://ci.centos.org/job/tripleo

Re: [rdo-dev] [infra][tripleoci] ppc64le container images in registry.rdoproject.org

[Wesley Hayutin]

On Thu, Mar 21, 2019 at 12:20 PM Trevor Vardeman 
wrote:

>
>
> > On Mar 21, 2019, at 1:12 PM, Wesley Hayutin  wrote:
> >
> >
> >
> > On Thu, Mar 21, 2019 at 12:03 PM Trevor Vardeman 
> wrote:
> > I've been working with mjturek and baha on this a bit.  I've responded
> inline below, but also want to clarify on the desired workflow.
> >
> > TL;DR: The desired workflow is to have ppc64le and x86_64 seamlessly
> integrated and uploaded.  This can be done with docker manifest list images.
> >
> > The following link explains in greater detail:
> https://docs.docker.com/registry/spec/manifest-v2-2/
> >
> > The process boils down to the following steps:
> >
> > 1) Upload an image of the first architecture  (ex:
> image1:x86_64_01012019)
> > 2) Upload an image of the second architecture  (ex:
> image1:ppc64le_01012019)
> > 3) Upload manifest list image of the image  (ex: image1:01012019)
> >
> >
> > I think this is implying we upload x86_64, and PPC containers to the rdo
> registry together.  We specifically do NOT want to do that.  One set of
> images should not be blocked by the other at this stage of the testing.
>
> That's not quite right, the images can be uploaded separately, but for the
> manifest list image to function properly you'd need both images uploaded.
> Also, as for the tagging, I wasn't suggesting a change to how images are
> tagged save for potentially modifying it such that the manifest list image
> is the default tag, versus the x86_64 image.
>

OK.. thanks Trevor for the explanation.  Since you are executing in
ci.centos and not via Zuul I can see how you are having to readdress a lot
of the infrastructure bits like uploading to the registry.

If we can get you on the zuul platform you won't have to rework or redo at
least some of that work. In zuul we have some post playbooks that execute
after the build job [1]. I think you'll be able to find just about
everything we do w/ containers here [2] now.

Perhaps you can join the tripleo community sync on Tuesdays directly after
the TripleO upstream community meeting on #tripleo so we can get you
introduced to some folks who may not have met you yet.

Thanks

[1]
https://github.com/openstack-infra/tripleo-ci/blob/master/playbooks/tripleo-buildcontainers/tag.yaml
[2]
https://github.com/openstack-infra/tripleo-ci/tree/master/playbooks/tripleo-buildcontainers

> >
> > I only just read through the manifest doc, so I may be misunderstanding
> something.
> > Any container built w/ tripleo should be tagged w/ the dlrn_hash used
> while building so if we needed some containers from x86_64 and some from
> PPC that should be relatively simple.
> >
> >
> > Step 3 is essentially just pushing a JSON body that has descriptors and
> references to the other two images, such that when someone does a pull
> request of the manifest list image, it will gather the appropriate
> architecture for that image based on the host's architecture.
> >
> > -Trevor
> >
> > PS. If I've missed something important with the overall concerns here I
> apologize, but thought it necessary to spell out the goal as I understand
> it.
> >
> > > On Mar 21, 2019, at 12:28 PM, Javier Pena  wrote:
> > >
> > >
> > > - Original Message -
> > >> Hi all,
> > >>
> > >> Over the last few weeks, mjturek and baha have been busy working on a
> set of
> > >> periodic jobs to build TripleO images for the ppc64le arch [1].
> > >>
> > >> The current missing step is publishing those images, and they are
> proposing
> > >> to push those to the RDO Registry instance at registry.rdoproject.org,
> just
> > >> like we do with our TripleO images. I have tried to understand the
> > >> requirements, and would like to get input on the following topics:
> > >>
> > >> - Which namespace would these images use? Based on some logs [2] it
> looks
> > >> like they use tripleomaster-ppc64le, will they also push the images
> to that
> > >> namespace?
> >
> > I have no experience in namespaces inside of a registry or how that
> differentiates images from one another, but the images should be pushed (in
> my opinion) to the same location in which the x86 images reside.
> >
> > >> - Could this create any conflicts with our current promotion pipeline?
> >
> > This should not cause conflicts in current promotion pipeline, as the
> process should be an extension to existing functionality.
> >
> > >> - Is registry.rdo the right place for thos

Re: [rdo-dev] [infra][tripleoci] ppc64le container images in registry.rdoproject.org

[Wesley Hayutin]

On Thu, Mar 21, 2019 at 12:03 PM Trevor Vardeman 
wrote:

> I've been working with mjturek and baha on this a bit.  I've responded
> inline below, but also want to clarify on the desired workflow.
>
> TL;DR: The desired workflow is to have ppc64le and x86_64 seamlessly
> integrated and uploaded.  This can be done with docker manifest list images.
>
> The following link explains in greater detail:
> https://docs.docker.com/registry/spec/manifest-v2-2/
>
> The process boils down to the following steps:
>
> 1) Upload an image of the first architecture  (ex: image1:x86_64_01012019)
> 2) Upload an image of the second architecture  (ex:
> image1:ppc64le_01012019)
> 3) Upload manifest list image of the image  (ex: image1:01012019)
>
>
I think this is implying we upload x86_64, and PPC containers to the rdo
registry together.  We specifically do NOT want to do that.  One set of
images should not be blocked by the other at this stage of the testing.

I only just read through the manifest doc, so I may be misunderstanding
something.
Any container built w/ tripleo should be tagged w/ the dlrn_hash used while
building so if we needed some containers from x86_64 and some from PPC that
should be relatively simple.



> Step 3 is essentially just pushing a JSON body that has descriptors and
> references to the other two images, such that when someone does a pull
> request of the manifest list image, it will gather the appropriate
> architecture for that image based on the host's architecture.
>
> -Trevor
>
> PS. If I've missed something important with the overall concerns here I
> apologize, but thought it necessary to spell out the goal as I understand
> it.
>
> > On Mar 21, 2019, at 12:28 PM, Javier Pena  wrote:
> >
> >
> > - Original Message -
> >> Hi all,
> >>
> >> Over the last few weeks, mjturek and baha have been busy working on a
> set of
> >> periodic jobs to build TripleO images for the ppc64le arch [1].
> >>
> >> The current missing step is publishing those images, and they are
> proposing
> >> to push those to the RDO Registry instance at registry.rdoproject.org,
> just
> >> like we do with our TripleO images. I have tried to understand the
> >> requirements, and would like to get input on the following topics:
> >>
> >> - Which namespace would these images use? Based on some logs [2] it
> looks
> >> like they use tripleomaster-ppc64le, will they also push the images to
> that
> >> namespace?
>
> I have no experience in namespaces inside of a registry or how that
> differentiates images from one another, but the images should be pushed (in
> my opinion) to the same location in which the x86 images reside.
>
> >> - Could this create any conflicts with our current promotion pipeline?
>
> This should not cause conflicts in current promotion pipeline, as the
> process should be an extension to existing functionality.
>
> >> - Is registry.rdo the right place for those images? I'm not familiar
> with the
> >> next steps for ppc64le images after that (will it then go through a
> >> promotion pipeline?), so that might affect.
>
> If the x86 images exist in registry.rdo, then the ppc64le (and any other
> architecture image) should exist there as well.  I can't think of a reason
> to differentiate between architectures when the desired result is parity
> and support of more architectures.
>
> >>
> >> If we decide to upload the images to images.rdo, we'll need to do the
> >
> > Correction: this should read "registry.rdo"
> >
> >> following:
> >>
> >> - Create the tripleomaster-ppc64le namespace in registry.rdo, following
> a
> >> similar pattern to [3].
> >> - Schedule a short registry downtime to increase its disk space, since
> it is
> >> currently near its limit.
>
> This is definitely necessary, given the capacity requirement will double,
> give or take, to support the additional architecture.
>
> >> - Update the job at ci.centos to include the REGISTRY_PASSWORD
> environment
> >> variable with the right token (see [4]). This is missing today, and
> causing
> >> the job failure.
> >>
> >> Once we get input from all interested parties, we will decide on the
> next
> >> steps.
> >>
> >> Thanks,
> >> Javier
> >>
> >>
> >> [1] -
> >>
> https://ci.centos.org/job/tripleo-upstream-containers-build-master-ppc64le/
> >> [2] -
> >>
> https://centos.logs.rdoproject.org/tripleo-upstream-containers-build-master-ppc64le/422/logs/logs/000_FAILED_tripleoclient.log
> >> [3] - https://review.rdoproject.org/r/19063
> >> [4] -
> >>
> https://github.com/rdo-infra/review.rdoproject.org-config/blob/master/playbooks/tripleo-ci-periodic-base/containers-build.yaml#L12-L20
> >> ___
> >> dev mailing list
> >> dev@lists.rdoproject.org
> >> http://lists.rdoproject.org/mailman/listinfo/dev
> >>
> >> To unsubscribe: dev-unsubscr...@lists.rdoproject.org
> >>
> > ___
> > dev mailing list
> > dev@lists.rdoproject.org
> > http://lists.rdoproject.org/mailman

Re: [rdo-dev] [infra][tripleoci] ppc64le container images in registry.rdoproject.org

[Wesley Hayutin]

On Thu, Mar 21, 2019 at 11:19 AM Javier Pena  wrote:

>
>
> - Original Message -
> > Hi all,
> >
> > Over the last few weeks, mjturek and baha have been busy working on a
> set of
> > periodic jobs to build TripleO images for the ppc64le arch [1].
> >
> > The current missing step is publishing those images, and they are
> proposing
> > to push those to the RDO Registry instance at registry.rdoproject.org,
> just
> > like we do with our TripleO images. I have tried to understand the
> > requirements, and would like to get input on the following topics:
> >
> > - Which namespace would these images use? Based on some logs [2] it looks
> > like they use tripleomaster-ppc64le, will they also push the images to
> that
> > namespace?
>

I've been warning my folks this was coming.
If the PPC folks are using  tripleomaster-ppc64le, I would propose we
update the x86_64 containers to be tripleo$release-x86_64


> > - Could this create any conflicts with our current promotion pipeline?
>

Only if we start to overload the rdo registry, which has happened in the
past. I don't see how the two would conflict otherwise


> > - Is registry.rdo the right place for those images? I'm not familiar
> with the
> > next steps for ppc64le images after that (will it then go through a
> > promotion pipeline?), so that might affect.
>

I think it's the right place to upload container images.
Overcloud images should be uploaded to https://images.rdoproject.org/master ,
but we're going to have to account for arch type there too.  Increase space
etc..

Finally they should build periodic jobs to build containers, overcloud
images and test jobs that trigger off of
https://trunk.rdoproject.org/centos7-master/tripleo-ci-testing/ and report
results to the dlrn_api.

If things proceed well and everyone has access to debug info and logs I
could see us adding ppc as promotion criteria.

Thanks for the update Javier!


> >
> > If we decide to upload the images to images.rdo, we'll need to do the
>
> Correction: this should read "registry.rdo"
>
> > following:
> >
> > - Create the tripleomaster-ppc64le namespace in registry.rdo, following a
> > similar pattern to [3].
> > - Schedule a short registry downtime to increase its disk space, since
> it is
> > currently near its limit.
> > - Update the job at ci.centos to include the REGISTRY_PASSWORD
> environment
> > variable with the right token (see [4]). This is missing today, and
> causing
> > the job failure.
> >
> > Once we get input from all interested parties, we will decide on the next
> > steps.
> >
> > Thanks,
> > Javier
> >
> >
> > [1] -
> >
> https://ci.centos.org/job/tripleo-upstream-containers-build-master-ppc64le/
> > [2] -
> >
> https://centos.logs.rdoproject.org/tripleo-upstream-containers-build-master-ppc64le/422/logs/logs/000_FAILED_tripleoclient.log
> > [3] - https://review.rdoproject.org/r/19063
> > [4] -
> >
> https://github.com/rdo-infra/review.rdoproject.org-config/blob/master/playbooks/tripleo-ci-periodic-base/containers-build.yaml#L12-L20
> > ___
> > dev mailing list
> > dev@lists.rdoproject.org
> > http://lists.rdoproject.org/mailman/listinfo/dev
> >
> > To unsubscribe: dev-unsubscr...@lists.rdoproject.org
> >
> ___
> dev mailing list
> dev@lists.rdoproject.org
> http://lists.rdoproject.org/mailman/listinfo/dev
>
> To unsubscribe: dev-unsubscr...@lists.rdoproject.org
>
___
dev mailing list
dev@lists.rdoproject.org
http://lists.rdoproject.org/mailman/listinfo/dev

To unsubscribe: dev-unsubscr...@lists.rdoproject.org

Re: [rdo-dev] Proposing Gabriele Cerami (panda) and Felix Enrique Llorente Pastora (quiquell) as core on review.rdoproject.org repos config and rdo-jobs

[Wesley Hayutin]

On Wed, Feb 27, 2019 at 11:46 AM Ronelle Landy  wrote:

> Hi,
>
> I'd like to propose Gabriele and Quique as new cores on the
> review.rdoproject.org repos config and rdo-jobs.
>
> Currently, the RDO-CI team has only two cores on these repos.
>
> Gabriele and Quique have both done foundation-level work in these repos
> and are active reviewers. They are both knowledgeable in the design and
> workings of TripleO jobs on Zuul and have contributed to much of our work.
>
> Their core input and oversight of config and rdo-jobs would be an asset.
>
> The vote will end on Monday, March 4th.
>
> Thanks,
> Ronelle
>

Thanks for the proposal Ronelle!
+1 from me!



> ___
> dev mailing list
> dev@lists.rdoproject.org
> http://lists.rdoproject.org/mailman/listinfo/dev
>
> To unsubscribe: dev-unsubscr...@lists.rdoproject.org
>
___
dev mailing list
dev@lists.rdoproject.org
http://lists.rdoproject.org/mailman/listinfo/dev

To unsubscribe: dev-unsubscr...@lists.rdoproject.org

Re: [openstack-dev] [tripleo] Proposing Enrique Llorente Pastora as a core reviewer for TripleO

[Wesley Hayutin]

On Thu, Nov 15, 2018 at 8:52 AM Sagi Shnaidman  wrote:

> Hi,
> I'd like to propose Quique (@quiquell) as a core reviewer for TripleO.
> Quique is actively involved in improvements and development of TripleO and
> TripleO CI. He also helps in other projects including but not limited to
> Infrastructure.
> He shows a very good understanding how TripleO and CI works and I'd like
> suggest him as core reviewer of TripleO in CI related code.
>
> Please vote!
> My +1 is here :)
>

+1 for tripleo-ci core, I don't think we're proposing tripleo core atm.
Thanks for proposing and sending this Sagi!


>
> Thanks
> --
> Best regards
> Sagi Shnaidman
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [tripleo] Ansible getting bumped up from 2.4 -> 2.6.6

[Wesley Hayutin]

Greetings,

Please be aware of the following patch [1].  This updates ansible in
queens, rocky, and stein.
 This was just pointed out to me, and I didn't see it coming so I thought
I'd email the group.

That is all, thanks

[1] https://review.rdoproject.org/r/#/c/14960
-- 

Wes Hayutin

Associate MANAGER

Red Hat



whayu...@redhat.comT: +1919 <+19197544114>4232509 IRC:  weshay


View my calendar and check my availability for meetings HERE

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo] gate issues please do not approve/recheck

[Wesley Hayutin]

Thanks Alex!

On Thu, Nov 1, 2018 at 10:27 AM Alex Schultz  wrote:

> Ok since the podman revert patche has been successfully merged and
> we've landed most of the non-voting scenario patches, it should be OK
> to restore/recheck.  It would be a good idea to prioritized things to
> land and if it's not critical, let's hold off on approving until we're
> sure the gate is much better.
>
> Thanks,
> -Alex
>
> On Wed, Oct 31, 2018 at 9:39 AM Alex Schultz  wrote:
> >
> > Hey folks,
> >
> > So we have identified an issue that has been causing a bunch of
> > failures and proposed a revert of our podman testing[0].  We have
> > cleared the gate and are asking that you not approve or recheck any
> > patches at this time.  We will let you know when it is safe to start
> > approving things.
> >
> > Thanks,
> > -Alex
> >
> > [0] https://review.openstack.org/#/c/614537/
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
-- 

Wes Hayutin

Associate MANAGER

Red Hat



whayu...@redhat.comT: +1919 <+19197544114>4232509 IRC:  weshay


View my calendar and check my availability for meetings HERE

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [tripleo] shutting down 3rd party TripleO CI for measurements

[Wesley Hayutin]

Greetings,

The TripleO-CI team would like to consider shutting down all the third
party check jobs running against TripleO projects in order to measure
results with and without load on the cloud for some amount of time.  I
suspect we would want to shut things down for roughly 24-48 hours.

If there are any strong objects please let us know.
Thank you
-- 

Wes Hayutin

Associate MANAGER

Red Hat



whayu...@redhat.comT: +1919 <+19197544114>4232509 IRC:  weshay


View my calendar and check my availability for meetings HERE

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo] reducing our upstream CI footprint

[Wesley Hayutin]

On Wed, Oct 31, 2018 at 11:21 AM Alex Schultz  wrote:

> Hey everyone,
>
> Based on previous emails around this[0][1], I have proposed a possible
> reducing in our usage by switching the scenario001--011 jobs to
> non-voting and removing them from the gate[2]. This will reduce the
> likelihood of causing gate resets and hopefully allow us to land
> corrective patches sooner.  In terms of risks, there is a risk that we
> might introduce breaking changes in the scenarios because they are
> officially non-voting, and we will still be gating promotions on these
> scenarios.  This means that if they are broken, they will need the
> same attention and care to fix them so we should be vigilant when the
> jobs are failing.
>
> The hope is that we can switch these scenarios out with voting
> standalone versions in the next few weeks, but until that I think we
> should proceed by removing them from the gate.  I know this is less
> than ideal but as most failures with these jobs in the gate are either
> timeouts or unrelated to the changes (or gate queue), they are more of
> hindrance than a help at this point.
>
> Thanks,
> -Alex
>

I think I also have to agree.
Having to deploy with containers, update containers and run with two nodes
is no longer a very viable option upstream.  It's not impossible but it
should be the exception and not the rule for all our jobs.

Thanks Alex



>
> [0]
> http://lists.openstack.org/pipermail/openstack-dev/2018-October/136141.html
> [1]
> http://lists.openstack.org/pipermail/openstack-dev/2018-October/135396.html
> [2]
> https://review.openstack.org/#/q/topic:reduce-tripleo-usage+(status:open+OR+status:merged)
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
-- 

Wes Hayutin

Associate MANAGER

Red Hat



whayu...@redhat.comT: +1919 <+19197544114>4232509 IRC:  weshay


View my calendar and check my availability for meetings HERE

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo][ci][upgrade] New jobs for tripleo Upgrade in the CI.

[Wesley Hayutin]

On Fri, Oct 12, 2018 at 5:10 AM Sofer Athlan-Guyot 
wrote:

> Hi,
>
> Testing and maintaining a green status for upgrade jobs within the 3h
> time limit has proven to be a very difficult job to say the least.
>

Indeed

>
> The net result has been: we don't have anything even touching the
> upgrade code in the CI.
>
> So during Denver PTG it has been decided to give up on running a full
> upgrade job during the 3h time limit and instead to focus on two
> complementary approach to at least touch the upgrade code:
>  1. run a standalone upgrade: this test the ansible upgrade playbook;
>  2. run a N->N upgrade; this test the upgrade python code;


> And here there are, still not merged but seen working:
>  - tripleo-ci-centos-7-standalone-upgrade:
>https://review.openstack.org/#/c/604706/
>  - tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades:
>https://review.openstack.org/#/c/607848/9
>
> The first is good to merge (but other could disagree), the second could
> be as well (but I tend to disagree :))
>
> The first leverage the standalone deployment and execute an standalone
> upgrade just after it.
>
> The limitation is that it only tests non-HA services (sorry pidone,
> cannot test ha in standalone) and only the upgrade_tasks (ie not any
> workflow related to the upgrade cli)
>

This can be augmented with 3rd party.  The pidone team and the ci team are
putting the final touches on a 3rd party job for HA services.  Looking
forward, I could see a 3rd party upgrade job that runs the pidone
verification tests.


>
> The main benefits here are:
>  - ~2h to run the upgrade, still a bit long but far away from the 3h
>time limit;
>  - we trigger a yum upgrade so that we can catch problems there as well;
>  - we test the standalone upgrade which is good in itself;
>  - composable role available (as in standalone/all-in-all deployment) so
>you can make a specific upgrade test for your project if it fits into
>the standalone constraint;
>

These are all huge benefits over the previous implementation that have been
made available to us via the standalone deployment

>
> For this last point, if standalone specific role eventually goes into
> project testing (nova, neutron ...), they could have as well a way to
> test upgrade tasks.  This would be a best case scenario.
>

!   woot !!!
This is a huge point that TripleO folks need to absorb!!
!   woot !!!

In the next several sprints the TripleO CI team will do our best to focus
on the standalone deployments to convert TripleO's upstream jobs over and
paving the way for other projects to start consuming it.  IMHO I would
think other projects would be *very* interested in testing an upgrade of
their individual component w/o all the noise of unrelated
services/components.


>
> Now, for the second point, the N->N upgrade.  Its "limitation" is that
> ... well it doesn't run a yum upgrade at all.  We start from master and
> run the upgrade to master.
>
> It's main benefit are:
>  - it takes ~2h20 to run, so well under the 3h time;
>  - tripleoclient upgrade code is run, which is one thing that the
>standalone ugprade cannot do.
>  - It also tend to exercise idempotency of all the tasks as it runs them
>on an already "upgraded" node;
>  - As added bonus, it could gate the tripleo-upgrade role as well as it
>definitively loads all of the role's tasks[1]
>
> For those that stayed with me to this point, I'm throwing another CI
> test that already proved useful already (caught errors), it's the
> ansible-lint test.  After a standalone deployment we just run
> ansible-lint on all playbook generated[2].
>

This is nice, thanks chem!


>
> It produces standalone_ansible_lint.log[3] in the working directory. It
> only takes a couple of minute to install ansible-lint and run it. It
> definitively gate against typos and the like. It touches hard to
> reach code as well, for instance the fast_forward tasks are linted.
> Still no pidone tasks in there but it could easily be added to a job
> that has HA tasks generated.
>
> Note that by default ansible-lint barks, as the generated playbooks hit
> several lintage problems, so only syntax errors and misnamed tasks or
> parameters are currently activated.  But all the lint problems are
> logged in the above file and can be fixed later on.  At which point we
> could activate full lint gating.
>
> Thanks for this long reading, any comments, shout of victory, cry of
> despair and reviews are welcomed.
>
> [1] but this has still to be investigated.
> [2] testing review https://review.openstack.org/#/c/604756/ and main code
> https://review.openstack.org/#/c/604757/
> [3] sample output http://paste.openstack.org/show/731960/
> --
> Sofer Athlan-Guyot
> chem on #freenode
> Upgrade DFG.
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.

Re: [openstack-dev] [tripleo][puppet] clearing the gate and landing patches to help CI

[Wesley Hayutin]

On Thu, Oct 4, 2018 at 8:29 AM Alex Schultz  wrote:

> And master is blocked again.  We need
> https://review.openstack.org/#/c/607952/
>
> Thanks,
> -Alex
>

I just got infra to put that patch on the top of the queue.
We also need https://review.openstack.org/#/c/607904/ reviewed ASAP

Thanks


>
> On Fri, Sep 28, 2018 at 9:02 AM Alex Schultz  wrote:
> >
> > Hey Folks,
> >
> > Currently the tripleo gate is at 21 hours and we're continue to have
> > timeouts and now scenario001/004 (in queens/pike) appear to be broken.
> > Additionally we've got some patches in puppet-openstack that we need
> > to land in order to resolve broken puppet unit tests which is
> > affecting both projects.
> >
> > Currently we need to wait for the following to land in puppet:
> >
> https://review.openstack.org/#/q/I4875b8bc8b2333046fc3a08b4669774fd26c89cb
> > https://review.openstack.org/#/c/605350/
> >
> > In tripleo we currently have not identified the root cause for any of
> > the timeout failures so I'd for us to work on that before trying to
> > land anything else because the gate resets are killing us and not
> > helping anything.  We have landed a few patches that have improved the
> > situation but we're still hitting issues.
> >
> > https://bugs.launchpad.net/tripleo/+bug/1795009 is the bug for the
> > scenario001/004 issues.  It appears that we're ending up with a newer
> > version of ansible on the system then what the packages provide. Still
> > working on figuring out where it's coming from.
> >
> > Please do not approve anything or recheck unless it's to address CI
> > issues at this time.
> >
> > Thanks,
> > -Alex
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
-- 

Wes Hayutin

Associate MANAGER

Red Hat



whayu...@redhat.comT: +1919 <+19197544114>4232509 IRC:  weshay


View my calendar and check my availability for meetings HERE

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] Zuul job backlog

[Wesley Hayutin]

On Fri, Sep 28, 2018 at 3:02 PM Matt Riedemann  wrote:

> On 9/28/2018 3:12 PM, Clark Boylan wrote:
> > I was asked to write a followup to this as the long Zuul queues have
> persisted through this week. Largely because the situation from last week
> hasn't changed much. We were down the upgraded cloud region while we worked
> around a network configuration bug, then once that was addressed we ran
> into neutron port assignment and deletion issues. We think these are both
> fixed and we are running in this region again as of today.
> >
> > Other good news is our classification rate is up significantly. We can
> use that information to go through the top identified gate bugs:
> >
> > Network Connectivity issues to test nodes [2]. This is the current top
> of the list, but I think its impact is relatively small. What is happening
> here is jobs fail to connect to their test nodes early in the pre-run
> playbook and then fail. Zuul will rerun these jobs for us because they
> failed in the pre-run step. Prior to zuulv3 we had nodepool run a ready
> script before marking test nodes as ready, this script would've caught and
> filtered out these broken network nodes early. We now notice them late
> during the pre-run of a job.
> >
> > Pip fails to find distribution for package [3]. Earlier in the week we
> had the in region mirror fail in two different regions for unrelated
> errors. These mirrors were fixed and the only other hits for this bug come
> from Ara which tried to install the 'black' package on python3.5 but this
> package requires python>=3.6.
> >
> > yum, no more mirrors to try [4]. At first glance this appears to be an
> infrastructure issue because the mirror isn't serving content to yum. On
> further investigation it turned out to be a DNS resolution issue caused by
> the installation of designate in the tripleo jobs. Tripleo is aware of this
> issue and working to correct it.
> >
> > Stackviz failing on py3 [5]. This is a real bug in stackviz caused by
> subunit data being binary not utf8 encoded strings. I've written a fix for
> this problem athttps://review.openstack.org/606184, but in doing so found
> that this was a known issue back in March and there was already a proposed
> fix,https://review.openstack.org/#/c/555388/3. It would be helpful if the
> QA team could care for this project and get a fix in. Otherwise, we should
> consider disabling stackviz on our tempest jobs (though the output from
> stackviz is often useful).
> >
> > There are other bugs being tracked by e-r. Some are bugs in the
> openstack software and I'm sure some are also bugs in the infrastructure. I
> have not yet had the time to work through the others though. It would be
> helpful if project teams could prioritize the debugging and fixing of these
> issues though.
> >
> > [2]http://status.openstack.org/elastic-recheck/gate.html#1793370
> > [3]http://status.openstack.org/elastic-recheck/gate.html#1449136
> > [4]http://status.openstack.org/elastic-recheck/gate.html#1708704
> > [5]http://status.openstack.org/elastic-recheck/gate.html#1758054
>
> Thanks for the update Clark.
>
> Another thing this week is the logstash indexing is behind by at least
> half a day. That's because workers were hitting OOM errors due to giant
> screen log files that aren't formatted properly so that we only index
> INFO+ level logs, and were instead trying to index the entire file,
> which some of which are 33MB *compressed*. So indexing of those
> identified problematic screen logs has been disabled:
>
> https://review.openstack.org/#/c/606197/
>
> I've reported bugs against each related project.
>
> --
>
> Thanks,
>
> Matt
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



Greetings Clark and all,
The TripleO team would like to announce a significant change to the
upstream CI the project has in place today.

TripleO can at times consume a large share of the compute resources [1]
provided by the OpenStack upstream infrastructure team and OpenStack
providers.  The TripleO project has a large code base and high velocity of
change which alone can tax the upstream CI system [3]. Additionally like
other projects the issue is particularly acute when gate jobs are reset at
a high rate.  Unlike most other projects in OpenStack, TripleO uses
multiple nodepool nodes in each job to more closely emulate customer like
deployments.  While using multiple nodes per job helps to uncover bugs
that are not found in other projects, the resources used, the run time of
each job, and usability have proven to be challenging.  It has been a
challenge to maintain job run times, quality and usability for TripleO and
a challenge for the infra team to provide the required compute resources
for the project.

A simplification of our upst

Re: [openstack-dev] [E] [tripleo] quickstart for humans

[Wesley Hayutin]

On Wed, Sep 5, 2018 at 5:08 PM Wesley Hayutin  wrote:

> On Wed, Sep 5, 2018 at 3:55 PM Gordon, Kent <
> kent.gor...@verizonwireless.com> wrote:
>
>>
>>
>> > -Original Message-
>> > From: Honza Pokorny [mailto:ho...@redhat.com]
>> > Sent: Thursday, August 30, 2018 9:28 AM
>> > To: OpenStack Development Mailing List (not for usage questions)
>> > 
>> > Subject: [E] [openstack-dev] [tripleo] quickstart for humans
>> >
>> > Hello!
>> >
>> > Over the last few months, it seems that tripleo-quickstart has evolved
>> into a
>> > CI tool.  It's primarily used by computers, and not humans.
>> > tripleo-quickstart is a helpful set of ansible playbooks, and a
>> collection of
>> > feature sets.  However, it's become less useful for setting up
>> development
>> > environments by humans.  For example, devmode.sh was recently
>> > deprecated without a user-friendly replacement. Moreover, during some
>> > informal irc conversations in #oooq, some developers even mentioned the
>> > plan to merge tripleo-quickstart and tripleo-ci.
>> >
>> > I think it would be beneficial to create a set of defaults for
>> tripleo-quickstart
>> > that can be used to spin up new environments; a set of defaults for
>> humans.
>> > This can either be a well-maintained script in tripleo-quickstart
>> itself, or a
>> > brand new project, e.g.
>> > tripleo-quickstart-humans.  The number of settings, knobs, and flags
>> should
>> > be kept to a minimum.
>> >
>> > This would accomplish two goals:
>> >
>> > 1.  It would bring uniformity to the team.  Each environment is
>> > installed the same way.  When something goes wrong, we can
>> > eliminate differences in setup when debugging.  This should save a
>> > lot of time.
>> >
>> > 2.  Quicker and more reliable environment setup.  If the set of defaults
>> > is used by many people, it should container fewer bugs because more
>> > people using something should translate into more bug reports, and
>> > more bug fixes.
>> >
>> > These thoughts are coming from the context of tripleo-ui development.  I
>> > need an environment in order to develop, but I don't necessarily always
>> care
>> > about how it's installed.  I want something that works for most
>> scenarios.
>> >
>> > What do you think?  Does this make sense?  Does something like this
>> already
>> > exist?
>> >
>> > Thanks for listening!
>> >
>> > Honza
>>
>> What is the recommended way to bring up a small POC of TripleO outside of
>> CI?
>>
>> Documentation suggests using quickstart
>>
>>
>> https://docs.openstack.org/tripleo-docs/latest/install/introduction/architecture.html
>>
>> "For development or proof of concept (PoC) environments, Quickstart can
>> also be used."
>>
>> Quickstart.sh outside of CI has been broken for a while.
>> It requires zuul cloner to work.
>>
>> https://bugs.launchpad.net/tripleo/+bug/1754498
>>
>>
> The issue described in bug [1] was caused by pip requirement install
> errors being swallowed up and not written to the console.
> TripleO-QuickStart-Extras was not pip installed due to previous errors, and
> that would cause quickstart-extras.yml to not be installed.
>
> The root cause of the failures is that pip install dependencies are not
> working as expected or in the same way without a http proxy  server.  This
> bug [1] should be closed, a RFE bug to ensure things work w/ a http proxy
> server should be opened.
>
> Please let me know if your work proves otherwise.
> Thank you!
>
> [1] https://bugs.launchpad.net/tripleo/+bug/1754498
>
>
>
I just launched an install with the following.

export WD=/var/tmp/test; ./quickstart.sh  --no-clone --release
tripleo-ci/master  --tags all --clean  --teardown all  -w $WD
whayutin-testbox

Where whayutin-testbox is my remote testbox, everything is working well atm
however there may be an issue w/ the bmc [1]

[1] https://bugs.launchpad.net/tripleo/+bug/1790969


>
>
>
>>
>>
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
> --
>
> Wes Hayutin
>

Re: [openstack-dev] [E] [tripleo] quickstart for humans

[Wesley Hayutin]

On Wed, Sep 5, 2018 at 3:55 PM Gordon, Kent 
wrote:

>
>
> > -Original Message-
> > From: Honza Pokorny [mailto:ho...@redhat.com]
> > Sent: Thursday, August 30, 2018 9:28 AM
> > To: OpenStack Development Mailing List (not for usage questions)
> > 
> > Subject: [E] [openstack-dev] [tripleo] quickstart for humans
> >
> > Hello!
> >
> > Over the last few months, it seems that tripleo-quickstart has evolved
> into a
> > CI tool.  It's primarily used by computers, and not humans.
> > tripleo-quickstart is a helpful set of ansible playbooks, and a
> collection of
> > feature sets.  However, it's become less useful for setting up
> development
> > environments by humans.  For example, devmode.sh was recently
> > deprecated without a user-friendly replacement. Moreover, during some
> > informal irc conversations in #oooq, some developers even mentioned the
> > plan to merge tripleo-quickstart and tripleo-ci.
> >
> > I think it would be beneficial to create a set of defaults for
> tripleo-quickstart
> > that can be used to spin up new environments; a set of defaults for
> humans.
> > This can either be a well-maintained script in tripleo-quickstart
> itself, or a
> > brand new project, e.g.
> > tripleo-quickstart-humans.  The number of settings, knobs, and flags
> should
> > be kept to a minimum.
> >
> > This would accomplish two goals:
> >
> > 1.  It would bring uniformity to the team.  Each environment is
> > installed the same way.  When something goes wrong, we can
> > eliminate differences in setup when debugging.  This should save a
> > lot of time.
> >
> > 2.  Quicker and more reliable environment setup.  If the set of defaults
> > is used by many people, it should container fewer bugs because more
> > people using something should translate into more bug reports, and
> > more bug fixes.
> >
> > These thoughts are coming from the context of tripleo-ui development.  I
> > need an environment in order to develop, but I don't necessarily always
> care
> > about how it's installed.  I want something that works for most
> scenarios.
> >
> > What do you think?  Does this make sense?  Does something like this
> already
> > exist?
> >
> > Thanks for listening!
> >
> > Honza
>
> What is the recommended way to bring up a small POC of TripleO outside of
> CI?
>
> Documentation suggests using quickstart
>
>
> https://docs.openstack.org/tripleo-docs/latest/install/introduction/architecture.html
>
> "For development or proof of concept (PoC) environments, Quickstart can
> also be used."
>
> Quickstart.sh outside of CI has been broken for a while.
> It requires zuul cloner to work.
>
> https://bugs.launchpad.net/tripleo/+bug/1754498
>
>
The issue described in bug [1] was caused by pip requirement install errors
being swallowed up and not written to the console.
TripleO-QuickStart-Extras was not pip installed due to previous errors, and
that would cause quickstart-extras.yml to not be installed.

The root cause of the failures is that pip install dependencies are not
working as expected or in the same way without a http proxy  server.  This
bug [1] should be closed, a RFE bug to ensure things work w/ a http proxy
server should be opened.

Please let me know if your work proves otherwise.
Thank you!

[1] https://bugs.launchpad.net/tripleo/+bug/1754498





>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
-- 

Wes Hayutin

Associate MANAGER

Red Hat



whayu...@redhat.comT: +1919 <+19197544114>4232509 IRC:  weshay


View my calendar and check my availability for meetings HERE

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo] quickstart for humans

[Wesley Hayutin]

On Fri, Aug 31, 2018 at 7:12 AM Steven Hardy  wrote:

> On Thu, Aug 30, 2018 at 3:28 PM, Honza Pokorny  wrote:
> > Hello!
> >
> > Over the last few months, it seems that tripleo-quickstart has evolved
> > into a CI tool.  It's primarily used by computers, and not humans.
> > tripleo-quickstart is a helpful set of ansible playbooks, and a
> > collection of feature sets.  However, it's become less useful for
> > setting up development environments by humans.  For example, devmode.sh
> > was recently deprecated without a user-friendly replacement. Moreover,
> > during some informal irc conversations in #oooq, some developers even
> > mentioned the plan to merge tripleo-quickstart and tripleo-ci.
>
> I was recently directed to the reproducer-quickstart.sh script that's
> written in the logs directory for all oooq CI jobs - does that help as
> a replacement for the previous devmode interface?
>
> Not that familiar with it myself but it seems to target many of the
> use-cases you mention e.g uniform reproducer for issues, potentially
> quicker way to replicate CI results?
>
> Steve
>
>
Thanks Honza and Steve for sharing.
Steve is correctly pointing out that reproducer scripts [1] are the
upgraded version of what was known as devmode.  There are two main goals we
are trying to achieve as a CI team with regards reproducing CI.

A.  Ensure that a developer can reproduce what is executed upstream step by
step as closely as is possible to deliver a 1:1 matching result
B.  Ensure the reliability of the local run is as close to the reliability
of the upstream check job as possible.

The older devmode scripts did a rather poor job at both A and B, where the
reproducer_script will actually execute the upstream CI workflow once an
environment is provisioned.  The results should be identical as long as
there are no yum, or other network related issues.

CI is a very opinionated realm of work, a point that Jirka makes quite
well.  We have to focus on goals that are clearly defined.  The long term
goal is make TripleO very easy to use and deploy, not just make
tripleo-quickstart easy to use.

The TripleO CI team is happy to help Honza or Jason stand up a tripleo job
against the tripleo-ui repo.  At which point you should have something
testing your changes and the scripts and tools to reproduce that job.  I
never like to see an upstream repo w/o any real CI running against it.

Thanks


[1]
https://docs.openstack.org/tripleo-docs/latest/contributor/reproduce-ci.html


> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
-- 

Wes Hayutin

Associate MANAGER

Red Hat



whayu...@redhat.comT: +1919 <+19197544114>4232509 IRC:  weshay


View my calendar and check my availability for meetings HERE

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO][kolla-ansible][DevStack][Tempest][openstack-ansible] Collaborate towards creating a unified ansible tempest role in openstack-ansible project

[Wesley Hayutin]

On Mon, Aug 27, 2018 at 12:43 PM Mohammed Naser  wrote:

> Hi Chandan,
>
> This is great, I added some more OSA-side comments, I'd love for us to
> find sometime to sit down to discuss this at the PTG.
>
> Thanks,
> Mohammed
>
> On Mon, Aug 27, 2018 at 12:39 PM, Jesse Pretorius
>  wrote:
> >>On 8/27/18, 7:33 AM, "Chandan kumar"  wrote:
> >
> >>I have summarized the problem statement and requirements on this
> etherpad [3].
> >>Feel free to add your requirements and questions for the same on the
> >>etherpad so that we can shape the unified ansible role in a better
> >>way.
> >
> >>Links:
> >>1.
> http://lists.openstack.org/pipermail/openstack-dev/2018-August/133119.html
> >>2. https://github.com/openstack/openstack-ansible-os_tempest
> >>3. https://etherpad.openstack.org/p/ansible-tempest-role
> >
> > Thanks for compiling this Chandan. I've added the really base
> requirements from an OSA standpoint that come to mind and a question that's
> been hanging in the recesses of my mind for a while.
> >
> >
>

Thanks Chandan for kicking this off!


> > 
> > Rackspace Limited is a company registered in England & Wales (company
> registered number 038
> 97010)
> whose registered office is at 5 Millington Road, Hyde Park Hayes, Middlesex
> UB3 4AZ. Rackspace Limited privacy policy can be viewed at
> www.rackspace.co.uk/legal/privacy-policy - This e-mail message may
> contain confidential or privileged information intended for the recipient.
> Any dissemination, distribution or copying of the enclosed material is
> prohibited. If you receive this transmission in error, please notify us
> immediately by e-mail at ab...@rackspace.com and delete the original
> message. Your cooperation is appreciated.
> >
> __
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe:
> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>
> --
> Mohammed Naser — vexxhost
> -
> D. 514-316-8872 <(514)%20316-8872>
> D. 800-910-1726 ext. 200 <(800)%20910-1726>
> E. mna...@vexxhost.com
> W. http://vexxhost.com
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
-- 

Wes Hayutin

Associate MANAGER

Red Hat



whayu...@redhat.comT: +1919 <+19197544114>4232509 IRC:  weshay


View my calendar and check my availability for meetings HERE

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo] The Weekly Owl - 29th Edition

[Wesley Hayutin]

On Fri, Aug 24, 2018 at 2:40 PM Emilien Macchi  wrote:

> Welcome to the twenty-ninthest edition of a weekly update in TripleO world!
> The goal is to provide a short reading (less than 5 minutes) to
> learnwhat's new this week.
> Any contributions and feedback are welcome.
> Link to the previous version:
> http://lists.openstack.org/pipermail/openstack-dev/2018-August/133094.html
>
> General announcements
> =
> +--> This week we released Rocky RC1, branched stable/rocky and unless
> there are critical bugs we'll call it our final stable release.
> +--> The team is preparing for the next PTG:
> https://etherpad.openstack.org/p/tripleo-ptg-stein
>
> CI status
> =
> +--> Sprint theme: Zuul v3 migration (
> https://trello.com/b/U1ITy0cu/tripleo-and-rdo-ci?menu=filter&filter=label:Sprint%2018%20CI
> )
> +--> The Ruck and Rover for this sprint  are Marios and Wes. Please tell
> them any CI issue.
>

It's actually Sorin and myself while Marios is on PTO.
Might as well take the opportunity to welcome Sorin to the TripleO team :))




> +--> Promotion on master is 11 days, 1 day on Rocky, 3 days on Queens, 3
> days on Pike and 1 days on Ocata.
>
> Upgrades
> =
> +--> Adding support for upgrades when OpenShift is deployed.
>
> Containers
> =
> +--> Efforts to support Podman tracked here:
> https://trello.com/b/S8TmOU0u/tripleo-podman
>
> config-download
> =
> +--> This squad is down and we move forward with the Edge squad.
>
> Edge
> =
> +--> New squad created by James:
> https://etherpad.openstack.org/p/tripleo-edge-squad-status (more to come)
>
> Integration
> =
> +--> No updates this week.
>
> UI/CLI
> =
> +--> No updates this week.
>
> Validations
> =
> +--> No updates this week, reviews are needed:
> https://etherpad.openstack.org/p/tripleo-validations-squad-status
>
> Networking
> =
> +--> Good progress on Ansible ML2 driver
>
> Workflows
> =
> +--> Planning Stein: better Ansible integration, UI convergence, etc.
>
> Security
> =
> +--> Working on SElinux for containers (related to podman integration
> mainly)
>
> Owl fact
> =
> "One single Owl can go fast. Multiple owls, together, can go far."
> Source: a mix of an African proverb and my Friday-afternoon imagination.
>
>
> Thank you all for reading and stay tuned!
> --
> Your fellow reporter, Emilien Macchi
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
-- 

Wes Hayutin

Associate MANAGER

Red Hat



whayu...@redhat.comT: +1919 <+19197544114>4232509 IRC:  weshay


View my calendar and check my availability for meetings HERE

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo] CI is blocked

[Wesley Hayutin]

On Wed, Aug 15, 2018 at 10:13 PM Wesley Hayutin  wrote:

> On Wed, Aug 15, 2018 at 7:13 PM Alex Schultz  wrote:
>
>> Please do not approve or recheck anything until further notice. We've
>> got a few issues that have basically broken all the jobs.
>>
>> https://bugs.launchpad.net/tripleo/+bug/1786764
>
>
fix posted: https://review.openstack.org/#/c/592577/


>
>> https://bugs.launchpad.net/tripleo/+bug/1787226
>
>
Dupe of 1786764 <https://bugs.launchpad.net/tripleo/+bug/1786764>


>
>> https://bugs.launchpad.net/tripleo/+bug/1787244
>
>
Fixed Released: https://review.openstack.org/592146


>
>> https://bugs.launchpad.net/tripleo/+bug/1787268
>
>
Proposed:
https://review.openstack.org/#/c/592233/
https://review.openstack.org/#/c/592275/



> https://bugs.launchpad.net/tripleo/+bug/1736950
>
> w
>

Will post a patch to skip the above tempest test.

Also the patch to re-enable build-test-packages, the code that injects your
change into a rpm is about to merge.
https://review.openstack.org/#/c/592218/

Thanks Steve, Alex, Jistr and others :)


>
>>
>> Thanks,
>> -Alex
>>
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
> --
>
> Wes Hayutin
>
> Associate MANAGER
>
> Red Hat
>
> <https://www.redhat.com/>
>
> whayu...@redhat.comT: +1919 <+19197544114>4232509 IRC:  weshay
> <https://red.ht/sig>
>
> View my calendar and check my availability for meetings HERE
> <https://calendar.google.com/calendar/b/1/embed?src=whayu...@redhat.com&ctz=America/New_York>
>
-- 

Wes Hayutin

Associate MANAGER

Red Hat

<https://www.redhat.com/>

whayu...@redhat.comT: +1919 <+19197544114>4232509 IRC:  weshay
<https://red.ht/sig>

View my calendar and check my availability for meetings HERE
<https://calendar.google.com/calendar/b/1/embed?src=whayu...@redhat.com&ctz=America/New_York>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo] CI is blocked

[Wesley Hayutin]

On Wed, Aug 15, 2018 at 7:13 PM Alex Schultz  wrote:

> Please do not approve or recheck anything until further notice. We've
> got a few issues that have basically broken all the jobs.
>
> https://bugs.launchpad.net/tripleo/+bug/1786764
> https://bugs.launchpad.net/tripleo/+bug/1787226
> https://bugs.launchpad.net/tripleo/+bug/1787244
> https://bugs.launchpad.net/tripleo/+bug/1787268


https://bugs.launchpad.net/tripleo/+bug/1736950

w

>
>
> Thanks,
> -Alex
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
-- 

Wes Hayutin

Associate MANAGER

Red Hat



whayu...@redhat.comT: +1919 <+19197544114>4232509 IRC:  weshay


View my calendar and check my availability for meetings HERE

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[rdo-dev] ship DLRN and rdopkg

[Wesley Hayutin]

Greetings,

Upstream TripleO does not use EPEL by design.  Currently the rpms to
install dlrn and rdopkg are shipped in EPEL.
Any thoughts on making this work so we can install the two rpms vs. pip
install?
Would it be possible to ship dlrn and rdopkg in the dlrn-deps repo?

https://bugs.launchpad.net/tripleo/+bug/1787244


-- 

Wes Hayutin

Associate MANAGER

Red Hat



whayu...@redhat.comT: +1919 <+19197544114>4232509 IRC:  weshay


View my calendar and check my availability for meetings HERE

___
dev mailing list
dev@lists.rdoproject.org
http://lists.rdoproject.org/mailman/listinfo/dev

To unsubscribe: dev-unsubscr...@lists.rdoproject.org

Re: [rdo-dev] fedora 28 cloud images

[Wesley Hayutin]

Thanks Paul!
I'll check it out.

On Tue, Aug 14, 2018 at 10:31 AM Paul Belanger 
wrote:

> On Mon, Aug 13, 2018 at 09:44:09PM -0600, Wesley Hayutin wrote:
> > On Mon, Aug 13, 2018 at 7:28 PM Paul Belanger 
> wrote:
> >
> > > On Mon, Aug 13, 2018 at 05:04:55PM -0600, Wesley Hayutin wrote:
> > > > The Cloud images provided by Fedora or fairly old [1], they were last
> > > > updated in April.
> > > > Can we please host an updated version of this image so package
> updates
> > > > don't take 20-30 minutes?
> > > > Maybe we can ask the Fedora folks for a respin or if needed host one
> > > > ourselves?
> > > >
> > > > Thanks!
> > > >
> > > > [1]
> > > >
> > >
> http://mirror.sfo12.us.leaseweb.net/fedora/linux/releases/28/Cloud/x86_64/images/
> > >
> > > Which jobs are using this image? In nodepool we rebuild the fedora
> > > images each day, to avoid manually refreshing them or 20mins of
> updates.
> > >
> > > - Paul
> > >
> >
> > Paul,
> > Are those images republished somewhere I can download them?
> > Maybe we can start doing that in RDO SF if the upstream fedora images are
> > not published upstream
> >
> Update images can be rebuild by using tools/build-images.sh[2]. However,
> we should also be building these images downstream in nodepool for
> rdoproject.org too.
>
> - Paul
>
> [2]
> http://git.openstack.org/cgit/openstack-infra/project-config/tree/tools/build-image.sh
>
-- 

Wes Hayutin

Associate MANAGER

Red Hat

<https://www.redhat.com/>

whayu...@redhat.comT: +1919 <+19197544114>4232509 IRC:  weshay
<https://red.ht/sig>

View my calendar and check my availability for meetings HERE
<https://calendar.google.com/calendar/b/1/embed?src=whayu...@redhat.com&ctz=America/New_York>
___
dev mailing list
dev@lists.rdoproject.org
http://lists.rdoproject.org/mailman/listinfo/dev

To unsubscribe: dev-unsubscr...@lists.rdoproject.org

Re: [rdo-dev] fedora 28 cloud images

[Wesley Hayutin]

On Mon, Aug 13, 2018 at 7:28 PM Paul Belanger  wrote:

> On Mon, Aug 13, 2018 at 05:04:55PM -0600, Wesley Hayutin wrote:
> > The Cloud images provided by Fedora or fairly old [1], they were last
> > updated in April.
> > Can we please host an updated version of this image so package updates
> > don't take 20-30 minutes?
> > Maybe we can ask the Fedora folks for a respin or if needed host one
> > ourselves?
> >
> > Thanks!
> >
> > [1]
> >
> http://mirror.sfo12.us.leaseweb.net/fedora/linux/releases/28/Cloud/x86_64/images/
>
> Which jobs are using this image? In nodepool we rebuild the fedora
> images each day, to avoid manually refreshing them or 20mins of updates.
>
> - Paul
>

Paul,
Are those images republished somewhere I can download them?
Maybe we can start doing that in RDO SF if the upstream fedora images are
not published upstream
-- 

Wes Hayutin

Associate MANAGER

Red Hat

<https://www.redhat.com/>

whayu...@redhat.comT: +1919 <+19197544114>4232509 IRC:  weshay
<https://red.ht/sig>

View my calendar and check my availability for meetings HERE
<https://calendar.google.com/calendar/b/1/embed?src=whayu...@redhat.com&ctz=America/New_York>
___
dev mailing list
dev@lists.rdoproject.org
http://lists.rdoproject.org/mailman/listinfo/dev

To unsubscribe: dev-unsubscr...@lists.rdoproject.org

[rdo-dev] fedora 28 cloud images

[Wesley Hayutin]

The Cloud images provided by Fedora or fairly old [1], they were last
updated in April.
Can we please host an updated version of this image so package updates
don't take 20-30 minutes?
Maybe we can ask the Fedora folks for a respin or if needed host one
ourselves?

Thanks!

[1]
http://mirror.sfo12.us.leaseweb.net/fedora/linux/releases/28/Cloud/x86_64/images/
-- 

Wes Hayutin

Associate MANAGER

Red Hat



whayu...@redhat.comT: +1919 <+19197544114>4232509 IRC:  weshay


View my calendar and check my availability for meetings HERE

___
dev mailing list
dev@lists.rdoproject.org
http://lists.rdoproject.org/mailman/listinfo/dev

To unsubscribe: dev-unsubscr...@lists.rdoproject.org

Re: [openstack-dev] [openstack-ansible][kolla-ansible][tripleo] ansible roles: where they live and what do they do

[Wesley Hayutin]

On Thu, Aug 9, 2018 at 5:33 PM Alex Schultz  wrote:

> On Thu, Aug 9, 2018 at 2:56 PM, Doug Hellmann 
> wrote:
> > Excerpts from Alex Schultz's message of 2018-08-09 14:31:34 -0600:
> >> Ahoy folks,
> >>
> >> I think it's time we come up with some basic rules/patterns on where
> >> code lands when it comes to OpenStack related Ansible roles and as we
> >> convert/export things. There was a recent proposal to create an
> >> ansible-role-tempest[0] that would take what we use in
> >> tripleo-quickstart-extras[1] and separate it for re-usability by
> >> others.   So it was asked if we could work with the openstack-ansible
> >> team and leverage the existing openstack-ansible-os_tempest[2].  It
> >> turns out we have a few more already existing roles laying around as
> >> well[3][4].
> >>
> >> What I would like to propose is that we as a community come together
> >> to agree on specific patterns so that we can leverage the same roles
> >> for some of the core configuration/deployment functionality while
> >> still allowing for specific project specific customization.  What I've
> >> noticed between all the project is that we have a few specific core
> >> pieces of functionality that needs to be handled (or skipped as it may
> >> be) for each service being deployed.
> >>
> >> 1) software installation
> >> 2) configuration management
> >> 3) service management
> >> 4) misc service actions
> >>
> >> Depending on which flavor of the deployment you're using, the content
> >> of each of these may be different.  Just about the only thing that is
> >> shared between them all would be the configuration management part.
> >
> > Does that make the 4 things separate roles, then? Isn't the role
> > usually the unit of sharing between playbooks?
> >
>
> It can be, but it doesn't have to be.  The problem comes in with the
> granularity at which you are defining the concept of the overall
> action.  If you want a role to encompass all that is "nova", you could
> have a single nova role that you invoke 5 different times to do the
> different actions during the overall deployment. Or you could create a
> role for nova-install, nova-config, nova-service, nova-cells, etc etc.
> I think splitting them out into their own role is a bit too much in
> terms of management.   In my particular openstack-ansible is already
> creating a role to manage "nova".  So is there a way that I can
> leverage part of their process within mine without having to duplicate
> it.  You can pull in the task files themselves from a different so
> technically I think you could define a ansible-role-tripleo-nova that
> does some include_tasks: ../../os_nova/tasks/install.yaml but then
> we'd have to duplicate the variables in our playbook rather than
> invoking a role with some parameters.
>
> IMHO this structure is an issue with the general sharing concepts of
> roles/tasks within ansible.  It's not really well defined and there's
> not really a concept of inheritance so I can't really extend your
> tasks with mine in more of a programming sense. I have to duplicate it
> or do something like include a specific task file from another role.
> Since I can't really extend a role in the traditional OO programing
> sense, I would like to figure out how I can leverage only part of it.
> This can be done by establishing ansible variables to trigger specific
> actions or just actually including the raw tasks themselves.   Either
> of these concepts needs some sort of contract to be established to the
> other won't get broken.   We had this in puppet via parameters which
> are checked, there isn't really a similar concept in ansible so it
> seems that we need to agree on some community established rules.
>
> For tripleo, I would like to just invoke the os_nova role and pass in
> like install: false, service: false, config_dir:
> /my/special/location/, config_data: {...} and it spit out the configs.
> Then my roles would actually leverage these via containers/etc.  Of
> course most of this goes away if we had a unified (not file based)
> configuration method across all services (openstack and non-openstack)
> but we don't. :D
>

I like your idea here Alex.
So having a role for each of these steps is too much management I agree,
however
establishing a pattern of using tasks for each step may be a really good
way to cleanly handle this.

Are you saying something like the following?

openstack-nova-role/
* * /tasks/
* * /tasks/install.yml
* * /tasks/service.yml
* */tasks/config.yml
* */taks/main.yml
---
# main.yml

include: install.yml
when: nova_install|bool

include: service.yml
when: nova_service|bool

include: config.yml
when: nova_config.yml
--

Interested in anything other than tags :)
Thanks



>
> Thanks,
> -Alex
>
> > Doug
> >
> >
> __
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe:
> op

[rdo-dev] proposing rfolco as core on rdoproject.org-config

[Wesley Hayutin]

Greetings,
Folco has been doing a lot of work to configure zuul properly across
multiple projects.
I'd like propose him as a core for the entire project or the bits to
configure jobs in rdo-sf.

WDYT?

Thanks

-- 

Wes Hayutin

Associate MANAGER

Red Hat



w hayu...@redhat.comT: +1919 <+19197544114>4232509
   IRC:  weshay


View my calendar and check my availability for meetings HERE

___
dev mailing list
dev@lists.rdoproject.org
http://lists.rdoproject.org/mailman/listinfo/dev

To unsubscribe: dev-unsubscr...@lists.rdoproject.org

Re: [openstack-dev] [tripleo] 3rd party ovb jobs are down

[Wesley Hayutin]

On Mon, Aug 6, 2018 at 5:55 PM Wesley Hayutin  wrote:

> On Mon, Aug 6, 2018 at 12:56 PM Wesley Hayutin 
> wrote:
>
>> Greetings,
>>
>> There is currently an unplanned outtage atm for the tripleo 3rd party OVB
>> based jobs.
>> We will contact the list when there are more details.
>>
>> Thank you!
>>
>
> OK,
> I'm going to call an end to the current outtage. We are closely monitoring
> the ovb 3rd party jobs.
> I'll called for the outtage when we hit [1].  Once I deleted the stack
> that moved teh HA routers to back_up state, the networking came back online.
>
> Additionally Kieran and I had to work through a number of instances that
> required admin access to remove.
> Once those resources  were cleaned up our CI tooling removed the rest of
> the stacks in delete_failed status.The stacks in delete_failed status
> were holding ip address that were causing new stacks to fail [2]
>
> There are still active issues that could cause OVB jobs to fail.
> This connection issues [3] was originaly thought to be DNS, however that
> turned out to not be the case.
> You may also see your job have a "node_failure" status, Paul has sent
> updates about this issue and is working on a patch and integration into rdo
> software factory.
>
> The CI team is close to including all the console logs into the regular
> job logs, however if needed atm they can be viewed at [5].
> We are also adding the bmc to the list of instances that we collect logs
> from.
>
> *To summarize* the most recent outtage was infra related and the errors
> were swallowed up in the bmc console log that at the time was not available
> to users.
>
> We continue to monitor that ovb jobs at http://cistatus.tripleo.org/
> The  legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master job
> is at a 53% pass rate, it needs to move to a > 85% pass rate to match other
> check jobs.
>
> Thanks all!
>

Following up,
 legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master job is at
a 78.6% pass rate today.   Certainly an improvement.

We had a quick sync meeting this morning w/ RDO-Cloud admins, tripleo and
infra folks.  There are two remaining issues.
There is an active issue w/ network connections, and an issue w/ instances
booting into node_failure status.   New issues
creep up all the time and we're actively monitoring those as well.  Still
shooting for 85% pass rate.

Thanks all



>
> [1] https://bugzilla.redhat.com/show_bug.cgi?id=1570136
> [2] http://paste.openstack.org/show/727444/
> [3] https://bugs.launchpad.net/tripleo/+bug/1785342
> [4] https://review.openstack.org/#/c/584488/
> [5] http://38.145.34.41/console-logs/?C=M;O=D
>
>
>
>
>
>
>>
>> --
>>
>> Wes Hayutin
>>
>> Associate MANAGER
>>
>> Red Hat
>>
>> <https://www.redhat.com/>
>>
>> w hayu...@redhat.comT: +1919 <+19197544114>
>> 4232509 IRC:  weshay
>> <https://red.ht/sig>
>>
>> View my calendar and check my availability for meetings HERE
>> <https://calendar.google.com/calendar/b/1/embed?src=whayu...@redhat.com&ctz=America/New_York>
>>
> --
>
> Wes Hayutin
>
> Associate MANAGER
>
> Red Hat
>
> <https://www.redhat.com/>
>
> w hayu...@redhat.comT: +1919 <+19197544114>
> 4232509 IRC:  weshay
> <https://red.ht/sig>
>
> View my calendar and check my availability for meetings HERE
> <https://calendar.google.com/calendar/b/1/embed?src=whayu...@redhat.com&ctz=America/New_York>
>
-- 

Wes Hayutin

Associate MANAGER

Red Hat

<https://www.redhat.com/>

w hayu...@redhat.comT: +1919 <+19197544114>4232509
   IRC:  weshay
<https://red.ht/sig>

View my calendar and check my availability for meetings HERE
<https://calendar.google.com/calendar/b/1/embed?src=whayu...@redhat.com&ctz=America/New_York>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo] 3rd party ovb jobs are down

[Wesley Hayutin]

On Mon, Aug 6, 2018 at 12:56 PM Wesley Hayutin  wrote:

> Greetings,
>
> There is currently an unplanned outtage atm for the tripleo 3rd party OVB
> based jobs.
> We will contact the list when there are more details.
>
> Thank you!
>

OK,
I'm going to call an end to the current outtage. We are closely monitoring
the ovb 3rd party jobs.
I'll called for the outtage when we hit [1].  Once I deleted the stack that
moved teh HA routers to back_up state, the networking came back online.

Additionally Kieran and I had to work through a number of instances that
required admin access to remove.
Once those resources  were cleaned up our CI tooling removed the rest of
the stacks in delete_failed status.The stacks in delete_failed status
were holding ip address that were causing new stacks to fail [2]

There are still active issues that could cause OVB jobs to fail.
This connection issues [3] was originaly thought to be DNS, however that
turned out to not be the case.
You may also see your job have a "node_failure" status, Paul has sent
updates about this issue and is working on a patch and integration into rdo
software factory.

The CI team is close to including all the console logs into the regular job
logs, however if needed atm they can be viewed at [5].
We are also adding the bmc to the list of instances that we collect logs
from.

*To summarize* the most recent outtage was infra related and the errors
were swallowed up in the bmc console log that at the time was not available
to users.

We continue to monitor that ovb jobs at http://cistatus.tripleo.org/
The  legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master job is
at a 53% pass rate, it needs to move to a > 85% pass rate to match other
check jobs.

Thanks all!

[1] https://bugzilla.redhat.com/show_bug.cgi?id=1570136
[2] http://paste.openstack.org/show/727444/
[3] https://bugs.launchpad.net/tripleo/+bug/1785342
[4] https://review.openstack.org/#/c/584488/
[5] http://38.145.34.41/console-logs/?C=M;O=D






>
> --
>
> Wes Hayutin
>
> Associate MANAGER
>
> Red Hat
>
> <https://www.redhat.com/>
>
> w hayu...@redhat.comT: +1919 <+19197544114>
> 4232509 IRC:  weshay
> <https://red.ht/sig>
>
> View my calendar and check my availability for meetings HERE
> <https://calendar.google.com/calendar/b/1/embed?src=whayu...@redhat.com&ctz=America/New_York>
>
-- 

Wes Hayutin

Associate MANAGER

Red Hat

<https://www.redhat.com/>

w hayu...@redhat.comT: +1919 <+19197544114>4232509
   IRC:  weshay
<https://red.ht/sig>

View my calendar and check my availability for meetings HERE
<https://calendar.google.com/calendar/b/1/embed?src=whayu...@redhat.com&ctz=America/New_York>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [rdo-dev] proposing Quique as ci-config core

[Wesley Hayutin]

Quique is now core on ci projects hosted in RDO's software factory.
To be more specific he has been added the rdo-ci-core group in
review.rdoproject.org.

Thank you

On Wed, Jul 25, 2018 at 6:28 AM Sagi Shnaidman  wrote:

> +1 of course!
>
> On Mon, Jul 16, 2018 at 5:04 PM, John Trowbridge  wrote:
>
>> On Fri, Jul 13, 2018 at 1:25 PM, Ronelle Landy  wrote:
>>
>>>
>>>
>>> On Fri, Jul 13, 2018 at 10:58 AM, Wesley Hayutin 
>>> wrote:
>>>
>>>> Greetings,
>>>>
>>>> I'd like to propose Quique as a core member to
>>>> https://github.com/rdo-infra/ci-config/
>>>> WDYT
>>>>
>>>
>>> +1 - Quique has done an awesome job - I'd definitely support him as
>>> another core.
>>>
>>>
>>>>
>>>> Thanks
>>>> --
>>>>
>>>> Wes Hayutin
>>>>
>>>> Associate MANAGER
>>>>
>>>> Red Hat
>>>>
>>>> <https://www.redhat.com/>
>>>>
>>>> w hayu...@redhat.comT: +1919 <+19197544114>
>>>> 4232509 IRC:  weshay
>>>> <https://red.ht/sig>
>>>>
>>>> View my calendar and check my availability for meetings HERE
>>>> <https://calendar.google.com/calendar/b/1/embed?src=whayu...@redhat.com&ctz=America/New_York>
>>>>
>>>
>>>
>>> ___
>>> dev mailing list
>>> dev@lists.rdoproject.org
>>> http://lists.rdoproject.org/mailman/listinfo/dev
>>>
>>> To unsubscribe: dev-unsubscr...@lists.rdoproject.org
>>>
>>>
>> +1
>>
>> ___
>> dev mailing list
>> dev@lists.rdoproject.org
>> http://lists.rdoproject.org/mailman/listinfo/dev
>>
>> To unsubscribe: dev-unsubscr...@lists.rdoproject.org
>>
>>
>
>
> --
> Best regards
> Sagi Shnaidman
> ___
> dev mailing list
> dev@lists.rdoproject.org
> http://lists.rdoproject.org/mailman/listinfo/dev
>
> To unsubscribe: dev-unsubscr...@lists.rdoproject.org
>
-- 

Wes Hayutin

Associate MANAGER

Red Hat

<https://www.redhat.com/>

w hayu...@redhat.comT: +1919 <+19197544114>4232509
   IRC:  weshay
<https://red.ht/sig>

View my calendar and check my availability for meetings HERE
<https://calendar.google.com/calendar/b/1/embed?src=whayu...@redhat.com&ctz=America/New_York>
___
dev mailing list
dev@lists.rdoproject.org
http://lists.rdoproject.org/mailman/listinfo/dev

To unsubscribe: dev-unsubscr...@lists.rdoproject.org

[openstack-dev] [tripleo] 3rd party ovb jobs are down

[Wesley Hayutin]

Greetings,

There is currently an unplanned outtage atm for the tripleo 3rd party OVB
based jobs.
We will contact the list when there are more details.

Thank you!

-- 

Wes Hayutin

Associate MANAGER

Red Hat



w hayu...@redhat.comT: +1919 <+19197544114>4232509
   IRC:  weshay


View my calendar and check my availability for meetings HERE

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo][ci][metrics] Stucked in the middle of work because of RDO CI

[Wesley Hayutin]

On Tue, Jul 31, 2018 at 7:41 AM Sagi Shnaidman  wrote:

> Hi, Martin
>
> I see master OVB jobs are passing now [1], please recheck.
>
> [1] http://cistatus.tripleo.org/
>

Things have improved and I see a lot of jobs passing however at the same
time I see too many jobs failing due to node_failures.  We are tracking the
data from [1].  Certainly the issue is NOT ideal for development and we
need to remain focused on improving the situation.

Thanks

[1] https://softwarefactory-project.io/zuul/api/tenant/rdoproject.org/builds



>
>
> On Tue, Jul 31, 2018 at 12:24 PM, Martin Magr  wrote:
>
>> Greetings guys,
>>
>>   it is pretty obvious that RDO CI jobs in TripleO projects are broken
>> [0]. Once Zuul CI jobs will pass would it be possible to have AMQP/collectd
>> patches ([1],[2],[3]) merged please even though the negative result of RDO
>> CI jobs? Half of the patches for this feature is merged and the other half
>> is stucked in this situation, were nobody reviews these patches, because
>> there is red -1. Those patches passed Zuul jobs several times already and
>> were manually tested too.
>>
>> Thanks in advance for consideration of this situation,
>> Martin
>>
>> [0]
>> https://trello.com/c/hkvfxAdX/667-cixtripleoci-rdo-software-factory-3rd-party-jobs-failing-due-to-instance-nodefailure
>> [1] https://review.openstack.org/#/c/578749
>> [2] https://review.openstack.org/#/c/576057/
>> [3] https://review.openstack.org/#/c/572312/
>>
>> --
>> Martin Mágr
>> Senior Software Engineer
>> Red Hat Czech
>>
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>
>
> --
> Best regards
> Sagi Shnaidman
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
-- 

Wes Hayutin

Associate MANAGER

Red Hat



w hayu...@redhat.comT: +1919 <+19197544114>4232509
   IRC:  weshay


View my calendar and check my availability for meetings HERE

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo] PTL non-candidacy

[Wesley Hayutin]

On Wed, Jul 25, 2018 at 9:24 AM Alex Schultz  wrote:

> Hey folks,
>
> So it's been great fun and we've accomplished much over the last two
> cycles but I believe it is time for me to step back and let someone
> else do the PTLing.  I'm not going anywhere so I'll still be around to
> focus on the simplification and improvements that TripleO needs going
> forward.  I look forwards to continuing our efforts with everyone.
>
> Thanks,
> -Alex
>

Thanks for all the hard work, long hours and leadership!
You have done a great job, congrats on a great cycle.

Thanks

>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
-- 

Wes Hayutin

Associate MANAGER

Red Hat



w hayu...@redhat.comT: +1919 <+19197544114>4232509
   IRC:  weshay


View my calendar and check my availability for meetings HERE

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [rdo-dev] Status of python3 PoC in RDO - 13-jul-2018

[Wesley Hayutin]

On Tue, Jul 17, 2018 at 4:57 PM Paul Belanger  wrote:

> On Tue, Jul 17, 2018 at 06:16:04PM +0200, Alfredo Moralejo Alonso wrote:
> > On Tue, Jul 17, 2018 at 5:38 PM, Alan Pevec  wrote:
> >
> > > > I think having the two separated images is the only way we can
> ensure we
> > > are not polluting the image in the initial phase with packages newer
> that
> > > in the stabilized repo.
> > >
> > > This should be a small list, are any of those actually included in the
> > > base image?
> > >
> >
> > Yes, the list is small but we can't be sure if it will change at some
> > point, and analysing if we are having one of those cases on each change
> is
> > too error prone, IMO.
> >
> >
> > > Alternatively, which jobs use "normal" f28 images, could we switch
> > > them to use "stabilized" f28 ?
> > >
> >
> > fedora-stable has the required packages to run and build python3
> packages.
> > Currently it's missing some requirements for the jobs running on fedora
> 28
> > image although it's something we could work on.
> >
> >
> > >
> > > Alan
> > >
> >
> > Maybe I'm underestimating the cost of maintaining two different images
> for
> > fedora, but my understanding is that the resources the extra image uses
> and
> > the effort to maintain is workable. Please, correct me if i'm wrong.
>
> I wouldn't say minimal, but there is a cost. With fedora-29 around the
> corner,
> it does mean we have 2 iamges to update now, over one. How long does this
> image
> need to live for?  When can we get to a point of just using the default
> fedora
> image?
>
> - Paul
>

Hey guys,
Slight tangent, but wondering how far out we are from the CI team having to
get involved here? Is there any planning required at this point?

Thanks



> ___
> dev mailing list
> dev@lists.rdoproject.org
> http://lists.rdoproject.org/mailman/listinfo/dev
>
> To unsubscribe: dev-unsubscr...@lists.rdoproject.org
>
-- 

Wes Hayutin

Associate MANAGER

Red Hat



w hayu...@redhat.comT: +1919 <+19197544114>4232509
   IRC:  weshay


View my calendar and check my availability for meetings HERE

___
dev mailing list
dev@lists.rdoproject.org
http://lists.rdoproject.org/mailman/listinfo/dev

To unsubscribe: dev-unsubscr...@lists.rdoproject.org

Re: [openstack-dev] [tripleo] Plan to switch the undercloud to be containerized by default

[Wesley Hayutin]

On Sun, Jul 15, 2018 at 4:38 PM Emilien Macchi  wrote:

> On Tue, Jul 10, 2018, 7:57 PM Emilien Macchi,  wrote:
>
>> with [tripleo] tag...
>>
>> On Tue, Jul 10, 2018 at 7:56 PM Emilien Macchi 
>> wrote:
>>
>>> This is an update on where things are regarding $topic, based on
>>> feedback I've got from the work done recently:
>>>
>>> 1) Switch --use-heat to take a boolean and deprecate it
>>>
>>> We still want to allow users to deploy non containerized underclouds, so
>>> we made this patch so they can use --use-heat=False:
>>> https://review.openstack.org/#/c/581467/
>>> Also https://review.openstack.org/#/c/581468 and
>>> https://review.openstack.org/581180 as dependencies
>>>
>>> 2) Configure CI jobs for containerized undercloud, except scenario001,
>>> 002 for timeout reasons (and figure out this problem in a parallel effort)
>>>
>>> https://review.openstack.org/#/c/575330
>>> https://review.openstack.org/#/c/579755
>>>
>>> 3) Switch tripleoclient to deploy by default a containerized undercloud
>>>
>>> https://review.openstack.org/576218
>>>
>>
> It merged today, hopefully all CI jobs (including promotion) will continue
> to run smoothly. Thanks everyone involved in this big effort!
>
>
>>> 4) Improve performances in general so scenario001/002 doesn't timeout
>>> when containerized undercloud is enabled
>>>
>>> https://review.openstack.org/#/c/581183 is the patch that'll enable the
>>> containerized undercloud
>>> https://review.openstack.org/#/c/577889/ is a patch that enables
>>> pipelining in ansible/quickstart, but more is about to come, I'll update
>>> the patches tonight.
>>>
>>
> These scenarios are still on the edge of a timeout, we'll keep working on
> those.
>
>
>>> 5) Cleanup quickstart to stop using use-heat except for fs003 (needed to
>>> disable containers, and keep coverage for non containerized undercloud)
>>>
>>> https://review.openstack.org/#/c/581534/
>>>
>>>
>>> Reviews are welcome, we aim to merge this work by milestone 3, in less
>>> than 2 weeks from now.
>>>
>>
> Since we merged the majority of the work, I think we can close the
> blueprint and not require FFE.
> Any feedback on that statement is welcome.
>
> Thanks,
> Emilien
>

Nice work Emiliien!!
Thanks


> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
-- 

Wes Hayutin

Associate MANAGER

Red Hat



w hayu...@redhat.comT: +1919 <+19197544114>4232509
   IRC:  weshay


View my calendar and check my availability for meetings HERE

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[rdo-dev] proposing Quique as ci-config core

[Wesley Hayutin]

Greetings,

I'd like to propose Quique as a core member to
https://github.com/rdo-infra/ci-config/
WDYT

Thanks
-- 

Wes Hayutin

Associate MANAGER

Red Hat



w hayu...@redhat.comT: +1919 <+19197544114>4232509
   IRC:  weshay


View my calendar and check my availability for meetings HERE

___
dev mailing list
dev@lists.rdoproject.org
http://lists.rdoproject.org/mailman/listinfo/dev

To unsubscribe: dev-unsubscr...@lists.rdoproject.org

[openstack-dev] [tripleo][ci] PTG Stein topics

[Wesley Hayutin]

Greetings,

Starting to collect thoughts and comments here,
https://etherpad.openstack.org/p/tripleoci-ptg-stein

Thanks
-- 

Wes Hayutin

Associate MANAGER

Red Hat



w hayu...@redhat.comT: +1919 <+19197544114>4232509
   IRC:  weshay


View my calendar and check my availability for meetings HERE

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo] TripleO CI squad status 7/03/2018

[Wesley Hayutin]

Apologies,
The dark theme in my browser changed the font color of the text.

The email is available at
http://lists.openstack.org/pipermail/openstack-dev/2018-July/131984.html
Thank you!


On Tue, Jul 3, 2018 at 7:13 PM Wesley Hayutin  wrote:

> Greetings
>
> The TripleO squad has just completed Sprint 15 (6-15 - 7/03).
> The following is a summary of activities during this sprint.
>
> Epic:
> # Sprint 15 Epic (CI Squad): Begin migration of upstream jobs to native
> zuulv3.
> For a list of the completed and remaining items for the sprint please
> refer to the following Epic card and the sub cards.
> https://trello.com/c/bQuQ9aWF/802-sprint-15-ci-goals
>
> Items to Note:
> * Timeouts in jobs are a recurring issue upstream.  How to handle and fix
> the timeouts is under discussion.  Note, containers may be contributing to
> the timeouts.
>
> Ruck / Rover:
>
> TripleO Master, 0 days since last promotion
> TripleO Queens, 2 days since last promotion
> TripleO Pike, 20 days since last promotion
>* This is failing in tempest and should be resolved with
> https://review.openstack.org/#/c/579937/
>
> https://review.rdoproject.org/etherpad/p/ruckrover-sprint15
>
>
> CRITICAL IN PROGRESS
> #1779561 No realm key for 'realm1'
> tripleo  Assignee: None  Reporter: wes hayutin  2 days old  Tags:
> promotion-blocker   6
> CRITICAL IN PROGRESS
> #1779271 periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset020-queens
> Details: volume c414293d-eb0f-4d74-8b4d-f9a15e23d399 failed to reach in-use
> status (current available) within the required time (500 s).
> tripleo  Assignee: yatin  Reporter: Quique Llorente  4 days old  Tags:
> promotion-blocker   14
> CRITICAL FIX RELEASED
> #1779263 "AnsibleUndefinedVariable: 'dict object' has no attribute
> 'overcloud'"} at
> periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset010-master
> tripleo  Assignee: Quique Llorente  Reporter: Quique Llorente  4 days old
> Tags: promotion-blocker   6
> CRITICAL FIX RELEASED
> #1778847 fs027 __init__() got an unexpected keyword argument 'cafile'
> tripleo  Assignee: wes hayutin  Reporter: Quique Llorente  6 days old
> Tags: promotion-blocker  quickstart  6
> CRITICAL FIX RELEASED
> #1778472 docker pull failed: Get
> https://registry-1.docker.io/v2/tripleomaster/centos-binary-rsyslog-base/manifests/current-tripleo:
> received unexpected HTTP status: 503 Service Unavailable
> tripleo  Assignee: Quique Llorente  Reporter: Quique Llorente  8 days old
> Tags: alert  ci  promotion-blocker   6
> CRITICAL FIX RELEASED
> #1778201 os-refresh-config undercloud install Error: Evaluation Error:
> Error while evaluating a Function Call, pick(): must receive at least one
> non empty
> tripleo  Assignee: Quique Llorente  Reporter: Quique Llorente  11 days
> old  Tags: ci  promotion-blocker   6
> CRITICAL FIX RELEASED
> #1778040 Error at overcloud_prep_containers Package:
> qpid-dispatch-router-0.8.0-1.el7.x86_64 (@delorean-master-testing)", "
> Requires: libqpid-proton.so.10()(64bit)
> tripleo  Assignee: Quique Llorente  Reporter: Quique Llorente  12 days old
> Tags: alert  ci  promotion-blocker  quickstart   10
> CRITICAL FIX RELEASED
> #159 pike, volume failed to build in error status. list index out of
> range in cinder
> tripleo  Assignee: wes hayutin  Reporter: wes hayutin  13 days old  Tags:
> alert  promotion-blocker   12
> CRITICAL FIX RELEASED
> #1777616 Undercloud installation is failing: Class[Neutron]: has no
> parameter named 'rabbit_hosts'
> tripleo  Assignee: yatin  Reporter: yatin  14 days old  Tags: alert
> promotion-blocker   6
> CRITICAL FIX RELEASED
> #1777541 undercloud install error, mistra 503 unavailable
> tripleo  Assignee: Alex Schultz  Reporter: wes hayutin  14 days old  Tags:
> alert  promotion-blocker   10
> CRITICAL FIX RELEASED
> #1777451 Error: /Stage[main]/Ceph::Rgw::Keystone::Auth/Keystone_role
> Duplicate entry found with name Member
> tripleo  Assignee: Quique Llorente  Reporter: wes hayutin  15 days old
> Tags: promotion-blocker   18
> CRITICAL FIX RELEASED
> #1777261 convert-overcloud-undercloud.yml fails on missing
> update_containers variable
> tripleo  Assignee: Sagi (Sergey) Shnaidman  Reporter: wes hayutin  17 days
> old  Tags: promotion-blocker   6
> CRITICAL FIX RELEASED
> #1777168 Failures to build python-networking-ovn
> tripleo  Assignee: Emilien Macchi  Reporter: Emilien Macchi  18 days old
> Tags: alert  ci  promotion-blocker   6
> CRITICAL FIX RELEASED
> #1777130 RDO cloud is down
> tripleo  Assignee: Quique Llorente  Reporter: Quique Llorente  18 days
> old  Tags: alert  promotion-blocker
> --
>
> Wes Hayutin
>
>

[openstack-dev] [tripleo] TripleO Tempest squad status 07/03/2018

[Wesley Hayutin]

Greetings,

The TripleO Tempest squad has just completed Sprint 15 (6-15 - 7/03).
The following is a summary of activities during this sprint.

Epic:
# Sprint 15 Epic ( Tempest Squad): Finish the refactoring of the core
OpenStack services in python-tempestconf, refstack certification tests, and
other miscellaneous work.  Chandan was on PTO most of the sprint, Arx was
active helping to resolve tempest issues, and Martin was focused on
refstack certifications so the progress was a little less than normal this
sprint.

For a list of the completed and remaining items for the sprint please refer
to the following Epic card and the sub cards.
https://trello.com/c/6QKG0HkU/801-sprint-15-python-tempestconf

Items to Note:
* Full runs of tempest are again fully passing in upstream master, queens.
Pike will be unblocked when https://review.openstack.org/#/c/579937/ merges.
* Chandan has volunteered to ruck / rove this sprint, so the team will
again be operating with only two active team members
* New documentation was created for containerized tempest
   *
https://docs.openstack.org/tripleo-docs/latest/install/basic_deployment/tempest.html#running-containerized-tempest-manually
   * Look for an upstream discussion around moving as much tempest
documentation to the tempest project as possible.
* Sprint 16 is the final sprint that will focus on refactoring
python-tempestconf.
-- 

Wes Hayutin

Associate MANAGER

Red Hat



w hayu...@redhat.comT: +1919 <+19197544114>4232509
   IRC:  weshay


View my calendar and check my availability for meetings HERE

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [tripleo] TripleO CI squad status 7/03/2018

[Wesley Hayutin]

Greetings

The TripleO squad has just completed Sprint 15 (6-15 - 7/03).
The following is a summary of activities during this sprint.

Epic:
# Sprint 15 Epic (CI Squad): Begin migration of upstream jobs to native
zuulv3.
For a list of the completed and remaining items for the sprint please refer
to the following Epic card and the sub cards.
https://trello.com/c/bQuQ9aWF/802-sprint-15-ci-goals

Items to Note:
* Timeouts in jobs are a recurring issue upstream.  How to handle and fix
the timeouts is under discussion.  Note, containers may be contributing to
the timeouts.

Ruck / Rover:

TripleO Master, 0 days since last promotion
TripleO Queens, 2 days since last promotion
TripleO Pike, 20 days since last promotion
   * This is failing in tempest and should be resolved with
https://review.openstack.org/#/c/579937/

https://review.rdoproject.org/etherpad/p/ruckrover-sprint15


CRITICAL IN PROGRESS
#1779561 No realm key for 'realm1'
tripleo  Assignee: None  Reporter: wes hayutin  2 days old  Tags:
promotion-blocker   6
CRITICAL IN PROGRESS
#1779271 periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset020-queens
Details: volume c414293d-eb0f-4d74-8b4d-f9a15e23d399 failed to reach in-use
status (current available) within the required time (500 s).
tripleo  Assignee: yatin  Reporter: Quique Llorente  4 days old  Tags:
promotion-blocker   14
CRITICAL FIX RELEASED
#1779263 "AnsibleUndefinedVariable: 'dict object' has no attribute
'overcloud'"} at
periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset010-master
tripleo  Assignee: Quique Llorente  Reporter: Quique Llorente  4 days old
Tags: promotion-blocker   6
CRITICAL FIX RELEASED
#1778847 fs027 __init__() got an unexpected keyword argument 'cafile'
tripleo  Assignee: wes hayutin  Reporter: Quique Llorente  6 days old
Tags: promotion-blocker  quickstart  6
CRITICAL FIX RELEASED
#1778472 docker pull failed: Get
https://registry-1.docker.io/v2/tripleomaster/centos-binary-rsyslog-base/manifests/current-tripleo:
received unexpected HTTP status: 503 Service Unavailable
tripleo  Assignee: Quique Llorente  Reporter: Quique Llorente  8 days old
Tags: alert  ci  promotion-blocker   6
CRITICAL FIX RELEASED
#1778201 os-refresh-config undercloud install Error: Evaluation Error:
Error while evaluating a Function Call, pick(): must receive at least one
non empty
tripleo  Assignee: Quique Llorente  Reporter: Quique Llorente  11 days old
Tags: ci  promotion-blocker   6
CRITICAL FIX RELEASED
#1778040 Error at overcloud_prep_containers Package:
qpid-dispatch-router-0.8.0-1.el7.x86_64 (@delorean-master-testing)", "
Requires: libqpid-proton.so.10()(64bit)
tripleo  Assignee: Quique Llorente  Reporter: Quique Llorente  12 days old
Tags: alert  ci  promotion-blocker  quickstart   10
CRITICAL FIX RELEASED
#159 pike, volume failed to build in error status. list index out of
range in cinder
tripleo  Assignee: wes hayutin  Reporter: wes hayutin  13 days old  Tags:
alert  promotion-blocker   12
CRITICAL FIX RELEASED
#1777616 Undercloud installation is failing: Class[Neutron]: has no
parameter named 'rabbit_hosts'
tripleo  Assignee: yatin  Reporter: yatin  14 days old  Tags: alert
promotion-blocker   6
CRITICAL FIX RELEASED
#1777541 undercloud install error, mistra 503 unavailable
tripleo  Assignee: Alex Schultz  Reporter: wes hayutin  14 days old  Tags:
alert  promotion-blocker   10
CRITICAL FIX RELEASED
#1777451 Error: /Stage[main]/Ceph::Rgw::Keystone::Auth/Keystone_role
Duplicate entry found with name Member
tripleo  Assignee: Quique Llorente  Reporter: wes hayutin  15 days old
Tags: promotion-blocker   18
CRITICAL FIX RELEASED
#1777261 convert-overcloud-undercloud.yml fails on missing
update_containers variable
tripleo  Assignee: Sagi (Sergey) Shnaidman  Reporter: wes hayutin  17 days
old  Tags: promotion-blocker   6
CRITICAL FIX RELEASED
#1777168 Failures to build python-networking-ovn
tripleo  Assignee: Emilien Macchi  Reporter: Emilien Macchi  18 days old
Tags: alert  ci  promotion-blocker   6
CRITICAL FIX RELEASED
#1777130 RDO cloud is down
tripleo  Assignee: Quique Llorente  Reporter: Quique Llorente  18 days old
Tags: alert  promotion-blocker
-- 

Wes Hayutin

Associate MANAGER

Red Hat



w hayu...@redhat.comT: +1919 <+19197544114>4232509
   IRC:  weshay


View my calendar and check my availability for meetings HERE

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [rdo-dev] zuulv3 migration update

[Wesley Hayutin]

On Thu, Jun 28, 2018 at 10:16 PM Paul Belanger 
wrote:

> On Mon, Jun 25, 2018 at 03:33:20PM -0400, Paul Belanger wrote:
> > On Fri, Jun 22, 2018 at 01:00:14PM -0400, Paul Belanger wrote:
> > > Greetings,
> > >
> > > Wanted to give an update to the zuulv3 migration process. As it stands
> right
> > > now, all projects specific to the rdoproject.org have been fully
> migrated from
> > > jenkins to zuulv3.  Hopefully this process was transparent to you, for
> the most
> > > part we were able to fully test everything before flipping the switch
> on a
> > > project.  However we did have a few issues with jobs, but thanks to
> jpena,
> > > amoralej and number80, to name a few, we managed to land fixes quickly.
> > >
> > > At this point, we are still waiting to complete the migration of
> tripleo-ci
> > > jobs, this include OVB and various 3rd party CI jobs.  We've been
> blocked by a
> > > nodepool migration of upstream-centos-7 nodes, but I'm told that will
> be
> > > finished today.  Meaning, we are aiming for Monday morning to flip the
> switch on
> > > the rest of the tripleo-ci specific projects.
> > >
> > > Due to this, we are still converting JJB changes into ansible
> playbooks using
> > > the zuul-migrate command, so I please hold of on moving forward with
> changes to
> > > the new rdo-jobs project.  If everything goes well on Monday, we'll
> remove the
> > > freeze on config / rdo-jobs for people to start contributing to again.
> > >
> > > Thanks for your patience during these last 3 weeks, and hopefully
> we'll be
> > > wrapped up Monday.  If you have any questions or problems, please
> reply here or
> > > #rdo on freenode.
> > >
> > Next update, today we successfully tested zuulv3 base jobs for devstack
> based
> > tripleo jobs.  This basically means we are ready to migrate the remaining
> > tripleo in rdoproject.org from jenkins to zuulv3. We still need to test
> OVB
> > jobs, but confident they will work.
> >
> > Nodepool capacity has been addressed however we are running into scale
> issues
> > with zuulv3 now. We have an IRC message out to the SoftwareFactory team
> to see
> > about bringing onlne more zuul-executors / zuul-mergers, hopefully this
> will not
> > take long.  Once that happens, I finally think we'll be ready to migrate
> the
> > left over tripleo jobs from jenkins to zuulv3.
> >
> Hopefully the last update before we make zuul migration complete.
>
> We've been able to validate tripleo OVB jobs are working correctly under
> zuulv3.
> Since it is now Friday, we won't migrate them from jenkins until early next
> week.
>
> Thanks again for the patience and look forward to sending the all clear
> next
> week.
>
> - Paul
>

Thanks Paul,
We have an odd message showing up in review.openstack from a third party
job atm [1].
Other than that I see things look good w/ third party jobs posting results
to reviews and the logs look good too.

Any idea what the following is caused by?
Thanks for the support!

RDO Third Party CI

Patch Set 4:

Merge Failed.

This change or one of its cross-repo dependencies was unable to be
automatically merged with the current state of its repository. Please
rebase the change and upload a new patchset.

[1]https://review.openstack.org/#/c/575862



>
>
> ___
> dev mailing list
> dev@lists.rdoproject.org
> http://lists.rdoproject.org/mailman/listinfo/dev
>
> To unsubscribe: dev-unsubscr...@lists.rdoproject.org
>
-- 

Wes Hayutin

Associate MANAGER

Red Hat



w hayu...@redhat.comT: +1919 <+19197544114>4232509
   IRC:  weshay


View my calendar and check my availability for meetings HERE

___
dev mailing list
dev@lists.rdoproject.org
http://lists.rdoproject.org/mailman/listinfo/dev

To unsubscribe: dev-unsubscr...@lists.rdoproject.org

Re: [rdo-dev] zuulv3 migration update

[Wesley Hayutin]

On Mon, Jun 25, 2018 at 3:33 PM Paul Belanger  wrote:

> On Fri, Jun 22, 2018 at 01:00:14PM -0400, Paul Belanger wrote:
> > Greetings,
> >
> > Wanted to give an update to the zuulv3 migration process. As it stands
> right
> > now, all projects specific to the rdoproject.org have been fully
> migrated from
> > jenkins to zuulv3.  Hopefully this process was transparent to you, for
> the most
> > part we were able to fully test everything before flipping the switch on
> a
> > project.  However we did have a few issues with jobs, but thanks to
> jpena,
> > amoralej and number80, to name a few, we managed to land fixes quickly.
> >
> > At this point, we are still waiting to complete the migration of
> tripleo-ci
> > jobs, this include OVB and various 3rd party CI jobs.  We've been
> blocked by a
> > nodepool migration of upstream-centos-7 nodes, but I'm told that will be
> > finished today.  Meaning, we are aiming for Monday morning to flip the
> switch on
> > the rest of the tripleo-ci specific projects.
> >
> > Due to this, we are still converting JJB changes into ansible playbooks
> using
> > the zuul-migrate command, so I please hold of on moving forward with
> changes to
> > the new rdo-jobs project.  If everything goes well on Monday, we'll
> remove the
> > freeze on config / rdo-jobs for people to start contributing to again.
> >
> > Thanks for your patience during these last 3 weeks, and hopefully we'll
> be
> > wrapped up Monday.  If you have any questions or problems, please reply
> here or
> > #rdo on freenode.
> >
> Next update, today we successfully tested zuulv3 base jobs for devstack
> based
> tripleo jobs.  This basically means we are ready to migrate the remaining
> tripleo in rdoproject.org from jenkins to zuulv3. We still need to test
> OVB
> jobs, but confident they will work.
>
> Nodepool capacity has been addressed however we are running into scale
> issues
> with zuulv3 now. We have an IRC message out to the SoftwareFactory team to
> see
> about bringing onlne more zuul-executors / zuul-mergers, hopefully this
> will not
> take long.  Once that happens, I finally think we'll be ready to migrate
> the
> left over tripleo jobs from jenkins to zuulv3.
>
> -Paul
>

Thanks Paul and to the other folks involved!



> ___
> dev mailing list
> dev@lists.rdoproject.org
> http://lists.rdoproject.org/mailman/listinfo/dev
>
> To unsubscribe: dev-unsubscr...@lists.rdoproject.org
>
-- 

Wes Hayutin

Associate MANAGER

Red Hat



w hayu...@redhat.comT: +1919 <+19197544114>4232509
   IRC:  weshay


View my calendar and check my availability for meetings HERE

___
dev mailing list
dev@lists.rdoproject.org
http://lists.rdoproject.org/mailman/listinfo/dev

To unsubscribe: dev-unsubscr...@lists.rdoproject.org

Re: [openstack-dev] [tripleo] CI is down stop workflowing

[Wesley Hayutin]

Check and gate jobs look clear.
More details on a bit.

Thanks

Sent from my mobile

On Tue, Jun 19, 2018, 07:33 Felix Enrique Llorente Pastora <
ellor...@redhat.com> wrote:

> Hi,
>
>We have the following bugs with fixes that need to land to unblock
> check/gate jobs:
>
>https://bugs.launchpad.net/tripleo/+bug/1777451
>https://bugs.launchpad.net/tripleo/+bug/1777616
>
>You can check them out at #tripleo ooolpbot.
>
>Please stop workflowing temporally until they get merged.
>
> BR.
>
> --
> Quique Llorente
>
> Openstack TripleO CI
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo] tripleo gate is blocked - please read

[Wesley Hayutin]

On Sat, Jun 16, 2018 at 10:21 AM Paul Belanger 
wrote:

> On Sat, Jun 16, 2018 at 12:47:10PM +, Jeremy Stanley wrote:
> > On 2018-06-15 23:15:01 -0700 (-0700), Emilien Macchi wrote:
> > [...]
> > > ## Dockerhub proxy issue
> > > Infra using wrong image layer object storage proxy for Dockerhub:
> > > https://review.openstack.org/#/c/575787/
> > > Huge thanks to infra team, specially Clark for fixing this super
> quickly,
> > > it clearly helped to stabilize our container jobs, I actually haven't
> seen
> > > timeouts since we merged your patch. Thanks a ton!
> > [...]
> >
> > As best we can tell from logs, the way Dockerhub served these images
> > changed a few weeks ago (at the end of May) leading to this problem.
> > --
> > Jeremy Stanley
>
> Should also note what we are doing here is a terrible hack, we've only
> been able
> to learn the information by sniffing the traffic to hub.docker.io for our
> reverse
> proxy cache configuration. It is also possible this can break in the
> future too,
> so something to always keep in the back of your mind.
>

Thanks Paul, Jeremy and the other infra folks involved.   The TripleO CI
team is working towards tracking the time on some of these container tasks
atm.  Thanks for doing what you guys could do given the circumstances.


>
> It would be great if docker tools just worked with HTTP proxies.
>
> -Paul
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [tripleo] zuul change gating repo name change

[Wesley Hayutin]

Greetings,

Please be aware the yum repo created in tripleo ci jobs is going to change
names to include the release [1].  This is done to ensure that only the
appropriate patches are installed when patches from multiple branches are
in play.  This is especially important to upgrade jobs.

If you are working on a patch that uses the gating.repo, this patch [1]
will impact your work.

Thank you!!

[1] https://review.openstack.org/#/c/572736/
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo] scenario000-multinode-oooq-container-upgrades

[Wesley Hayutin]

On Tue, Jun 12, 2018 at 11:21 AM James Slagle 
wrote:

> On Tue, Jun 12, 2018 at 11:03 AM, Jiří Stránský  wrote:
> > On 12.6.2018 15:06, James Slagle wrote:
> >>
> >> On Mon, Jun 11, 2018 at 3:34 PM, Wesley Hayutin 
> >> wrote:
> >>>
> >>> Greetings,
> >>>
> >>> I wanted to let everyone know that we have a keystone only deployment
> and
> >>> upgrade job in check non-voting.  I'm asking everyone in TripleO to be
> >>> mindful of this job and to help make sure it continues to pass as we
> move
> >>> it
> >>> from non-voting check to check and eventually gating.
> >>
> >>
> >> +1, nice work!
> >>
> >>> Upgrade jobs are particularly difficult to keep running successfully
> >>> because
> >>> of the complex workflow itself, job run times and other factors.  Your
> >>> help
> >>> to ensure we don't merge w/o a pass on this job will go a long way in
> >>> helping the tripleo upgrades team.
> >>>
> >>> There is still work to be done here, however it's much easier to do it
> >>> with
> >>> the check non-voting job in place.
> >>
> >>
> >> The job doesn't appear to be passing at all on stable/queens. I see
> >> this same failure on several patches:
> >>
> >>
> http://logs.openstack.org/59/571459/1/check/tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades/8bbd827/logs/undercloud/home/zuul/overcloud_upgrade_run_Controller.log.txt.gz
> >>
> >> Is this a known issue?
> >
> >
> > I think so, or to put it precisely, i only ever looked into making the
> job
> > work for master (and beyond).
> >
> > We could look into making it work on Queens too, but personally i think
> > effort would be better spent elsewhere at this point. E.g. upd+upg jobs
> with
> > more complete of services utilizing containerized undercloud (those would
> > not validate OC workflow at all, but would give coverage for
> > update_tasks/upgrade_tasks), user and dev docs around all lifecycle ops
> > (upd, upg, ffwd), upgrade work in the area of TLS by default, upgrade
> > handling for external_deploy_tasks (= "how do we upgrade Ceph in Rocky"),
> > also perhaps trying to DRY repeated parts of upgrade templates, etc.
> >
> > If someone wants to step up to iron out Queens issues with that job then
> we
> > can do it, but my 2 cents would be just to disable the job on Queens and
> > focus on the future.
>
> Sure, I'm just trying to figure out what can safely be ignored. The
> tone of the original email was encouraging reviewers not to ignore the
> job. Let's remove it from queens then, as right now it's just noise.
>

I think we missed a patch [1] to correctly set the release for the job.
I'll take a look at the results.

I may have jumped the gun w/ the tone of the email w/ regards to keeping it
running.  I'll make the adjustment on queens for now [2].

Thanks for catching that James, Jirka!

[1] https://review.openstack.org/#/c/574417/
[2] https://review.openstack.org/574794


>
>
>
> --
> -- James Slagle
> --
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [tripleo] scenario000-multinode-oooq-container-upgrades

[Wesley Hayutin]

Greetings,

I wanted to let everyone know that we have a keystone only deployment and
upgrade job in check non-voting.  I'm asking everyone in TripleO to be
mindful of this job and to help make sure it continues to pass as we move
it from non-voting check to check and eventually gating.

Upgrade jobs are particularly difficult to keep running successfully
because of the complex workflow itself, job run times and other factors.
Your help to ensure we don't merge w/o a pass on this job will go a long
way in helping the tripleo upgrades team.

There is still work to be done here, however it's much easier to do it with
the check non-voting job in place.

Thanks all
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [rdo-dev] Bringing OpenStack back to Fedora

[Wesley Hayutin]

On Fri, May 25, 2018 at 9:06 AM Neal Gompa  wrote:

> Hey all,
>
> It's been several years since OpenStack has been part of the Fedora
> distribution. Since then, a number of things have changed.
>
> Some of the major ones:
> * Python packaging has gotten better and easier in Fedora (the changes are
> numerous)
> * RDO now has Software Factory and full-range CI capabilities
> * Fedora has CI hook support for packaging (with the transition to Pagure
> dist-git)
> * OpenStack and Fedora schedules line up again:
> https://releases.openstack.org/
>
> In my view, it's a huge shame that we don't offer OpenStack for people to
> use on Fedora, integrated with the latest software. OpenStack distribution
> on other distros are able to pull this off, and I feel like we should be
> able to as well.
>
> So what I want to know is the following:
> * What are the (real or perceived) difficulties in packaging OpenStack for
> Fedora?
> * How difficult would it be to adapt RDO CI tooling to plug into Fedora
> Dist-Git?
> * How many of the underlying dependencies exist in Fedora today that were
> forked into CentOS for RDO and which ones?
> * What dependencies are in RDO that don't exist in Fedora?
>
> I'm willing to help with a lot of the packaging stuff, including adapting
> packages for Fedora, and helping with reviews to bring stuff back in. I
> firmly believe that maintenance of OpenStack in Fedora should be far easier
> than it was three years ago, when it was ripped out.
>

Hey Neal,
There is actually some planning and discussions around RDO and Fedora going
on atm.  I suspect there will be some updates to this list in the near
future.

Thanks


>
> So help me... Help you!
>
> Best regards,
> Neal
>
> --
> 真実はいつも一つ！/ Always, there's only one truth!
> ___
> dev mailing list
> dev@lists.rdoproject.org
> http://lists.rdoproject.org/mailman/listinfo/dev
>
> To unsubscribe: dev-unsubscr...@lists.rdoproject.org
>
___
dev mailing list
dev@lists.rdoproject.org
http://lists.rdoproject.org/mailman/listinfo/dev

To unsubscribe: dev-unsubscr...@lists.rdoproject.org

Re: [openstack-dev] [tripleo] Status of Standalone installer (aka All-In-One)

[Wesley Hayutin]

On Tue, Jun 5, 2018 at 3:31 AM Raoul Scarazzini  wrote:

> On 05/06/2018 02:26, Emilien Macchi wrote:
> [...]
> > I hope this update was useful, feel free to give feedback or ask any
> > questions,
> [...]
>
> I'm no prophet here, but I see a bright future for this approach. I can
> imagine how useful this can be on the testing and much more the learning
> side. Thanks for sharing!
>
> --
> Raoul Scarazzini
> ra...@redhat.com


Real big +1 to everyone who has contributed to the standalone
installer.
>From an end user experience, this is simple, fast! This is going to be the
base for some really cool work.

Emilien, the CI is working, enjoy your PTO :)
http://logs.openstack.org/17/572217/6/check/tripleo-ci-centos-7-standalone/b2eb1b7/logs/ara_oooq/result/bb49965e-4fb7-43ea-a9e3-c227702c17de/

Thanks!



>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [docs] Automating documentation the tripleo way?

[Wesley Hayutin]

On Thu, May 17, 2018 at 10:22 AM Petr Kovar  wrote:

> On Wed, 16 May 2018 13:26:46 -0600
> Wesley Hayutin  wrote:
>
> > On Wed, May 16, 2018 at 3:05 PM Doug Hellmann 
> wrote:
> >
> > > Excerpts from Wesley Hayutin's message of 2018-05-16 12:51:25 -0600:
> > > > On Wed, May 16, 2018 at 2:41 PM Doug Hellmann  >
> > > wrote:
> > > >
> > > > > Excerpts from Petr Kovar's message of 2018-05-16 17:39:14 +0200:
> > > > > > Hi all,
> > > > > >
> > > > > > In the past few years, we've seen several efforts aimed at
> automating
> > > > > > procedural documentation, mostly centered around the OpenStack
> > > > > > installation guide. This idea to automatically produce and verify
> > > > > > installation steps or similar procedures was mentioned again at
> the
> > > last
> > > > > > Summit (
> https://etherpad.openstack.org/p/SYD-install-guide-testing).
> > > > > >
> > > > > > It was brought to my attention that the tripleo team has been
> > > working on
> > > > > > automating some of the tripleo deployment procedures, using a
> Bash
> > > script
> > > > > > with included comment lines to supply some RST-formatted
> narrative,
> > > for
> > > > > > example:
> > > > > >
> > > > > >
> > > > >
> > >
> https://github.com/openstack/tripleo-quickstart-extras/blob/master/roles/overcloud-prep-images/templates/overcloud-prep-images.sh.j2
> > > > > >
> > > > > > The Bash script can then be converted to RST, e.g.:
> > > > > >
> > > > > >
> > > > >
> > >
> https://thirdparty.logs.rdoproject.org/jenkins-tripleo-quickstart-queens-rdo_trunk-baremetal-dell_fc430_envB-single_nic_vlans-27/docs/build/
> > > > > >
> > > > > > Source Code:
> > > > > >
> > > > > >
> > > > >
> > >
> https://github.com/openstack/tripleo-quickstart-extras/tree/master/roles/collect-logs
> > > > > >
> > > > > > I really liked this approach and while I don't want to sound like
> > > selling
> > > > > > other people's work, I'm wondering if there is still an interest
> > > among
> > > > > the
> > > > > > broader OpenStack community in automating documentation like
> this?
> > > > > >
> > > > > > Thanks,
> > > > > > pk
> > > > > >
> > > > >
> > > > > Weren't the folks doing the training-labs or training-guides
> taking a
> > > > > similar approach? IIRC, they ended up implementing what amounted to
> > > > > their own installer for OpenStack, and then ended up with all of
> the
> > > > > associated upgrade and testing burden.
> > > > >
> > > > > I like the idea of trying to use some automation from this, but I
> > > wonder
> > > > > if we'd be better off extracting data from other tools, rather than
> > > > > building a new one.
> > > > >
> > > > > Doug
> > > > >
> > > >
> > > > So there really isn't anything new to create, the work is done and
> > > executed
> > > > on every tripleo change that runs in rdo-cloud.
> > >
> > > It wasn't clear what Petr was hoping to get. Deploying with TripleO is
> > > only one way to deploy, so we wouldn't be able to replace the current
> > > installation guides with the results of this work. It sounds like
> that's
> > > not the goal, though.
>
>
> Yes, I wasn't very clear on the goals as I didn't want to make too many
> assumptions before learning about technical details from other people.
> Ben's comments made me realize this approach would probably be best suited
> for generating documents such as quick start guides or tutorials that are
> procedural, yet they don't aim at describing multiple use cases.
>
>
> > > >
> > > > Instead of dismissing the idea upfront I'm more inclined to set an
> > > > achievable small step to see how well it works.  My thought would be
> to
> > > > focus on the upcoming all-in-one installer and the automated doc
> > > generated
> > > > with tha

Re: [openstack-dev] [docs] Automating documentation the tripleo way?

[Wesley Hayutin]

On Wed, May 16, 2018 at 3:05 PM Doug Hellmann  wrote:

> Excerpts from Wesley Hayutin's message of 2018-05-16 12:51:25 -0600:
> > On Wed, May 16, 2018 at 2:41 PM Doug Hellmann 
> wrote:
> >
> > > Excerpts from Petr Kovar's message of 2018-05-16 17:39:14 +0200:
> > > > Hi all,
> > > >
> > > > In the past few years, we've seen several efforts aimed at automating
> > > > procedural documentation, mostly centered around the OpenStack
> > > > installation guide. This idea to automatically produce and verify
> > > > installation steps or similar procedures was mentioned again at the
> last
> > > > Summit (https://etherpad.openstack.org/p/SYD-install-guide-testing).
> > > >
> > > > It was brought to my attention that the tripleo team has been
> working on
> > > > automating some of the tripleo deployment procedures, using a Bash
> script
> > > > with included comment lines to supply some RST-formatted narrative,
> for
> > > > example:
> > > >
> > > >
> > >
> https://github.com/openstack/tripleo-quickstart-extras/blob/master/roles/overcloud-prep-images/templates/overcloud-prep-images.sh.j2
> > > >
> > > > The Bash script can then be converted to RST, e.g.:
> > > >
> > > >
> > >
> https://thirdparty.logs.rdoproject.org/jenkins-tripleo-quickstart-queens-rdo_trunk-baremetal-dell_fc430_envB-single_nic_vlans-27/docs/build/
> > > >
> > > > Source Code:
> > > >
> > > >
> > >
> https://github.com/openstack/tripleo-quickstart-extras/tree/master/roles/collect-logs
> > > >
> > > > I really liked this approach and while I don't want to sound like
> selling
> > > > other people's work, I'm wondering if there is still an interest
> among
> > > the
> > > > broader OpenStack community in automating documentation like this?
> > > >
> > > > Thanks,
> > > > pk
> > > >
> > >
> > > Weren't the folks doing the training-labs or training-guides taking a
> > > similar approach? IIRC, they ended up implementing what amounted to
> > > their own installer for OpenStack, and then ended up with all of the
> > > associated upgrade and testing burden.
> > >
> > > I like the idea of trying to use some automation from this, but I
> wonder
> > > if we'd be better off extracting data from other tools, rather than
> > > building a new one.
> > >
> > > Doug
> > >
> >
> > So there really isn't anything new to create, the work is done and
> executed
> > on every tripleo change that runs in rdo-cloud.
>
> It wasn't clear what Petr was hoping to get. Deploying with TripleO is
> only one way to deploy, so we wouldn't be able to replace the current
> installation guides with the results of this work. It sounds like that's
> not the goal, though.
>
> >
> > Instead of dismissing the idea upfront I'm more inclined to set an
> > achievable small step to see how well it works.  My thought would be to
> > focus on the upcoming all-in-one installer and the automated doc
> generated
> > with that workflow.  I'd like to target publishing the all-in-one tripleo
> > installer doc to [1] for Stein and of course a section of tripleo.org.
>
> As an official project, why is TripleO still publishing docs to its own
> site? That's not something we generally encourage.
>
> That said, publishing a new deployment guide based on this technique
> makes sense in general. What about Ben's comments elsewhere in the
> thread?
>

I think Ben is referring to an older implementation and a slightly
different design but still has some points that we would want to be mindful
of.   I think this is a worthy effort to take another pass at this
regarless to be honest as we've found a good combination of interested
folks and sometimes the right people make all the difference.

My personal opinion is that I'm not expecting the automated doc generation
to be upload ready to a doc server after each run.  I do expect it to do
95% of the work, and to help keep the doc up to date with what is executed
in the latest releases of TripleO.  Also noting the doc used is a mixture
of static and generated documentation which I think worked out quite well
in order to not soley rely on what is executed in ci.

So again, my thought is to create a small achievable goal and see where the
collaboration takes us.

Thanks


>
> Doug
>
> >
> > What do you think?
> >
> > [1] https://docs.openstack.org/queens/deploy/
> >
> > >
> > >
> __
> > > OpenStack Development Mailing List (not for usage questions)
> > > Unsubscribe:
> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> > >
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing

Re: [openstack-dev] [docs] Automating documentation the tripleo way?

[Wesley Hayutin]

On Wed, May 16, 2018 at 2:41 PM Doug Hellmann  wrote:

> Excerpts from Petr Kovar's message of 2018-05-16 17:39:14 +0200:
> > Hi all,
> >
> > In the past few years, we've seen several efforts aimed at automating
> > procedural documentation, mostly centered around the OpenStack
> > installation guide. This idea to automatically produce and verify
> > installation steps or similar procedures was mentioned again at the last
> > Summit (https://etherpad.openstack.org/p/SYD-install-guide-testing).
> >
> > It was brought to my attention that the tripleo team has been working on
> > automating some of the tripleo deployment procedures, using a Bash script
> > with included comment lines to supply some RST-formatted narrative, for
> > example:
> >
> >
> https://github.com/openstack/tripleo-quickstart-extras/blob/master/roles/overcloud-prep-images/templates/overcloud-prep-images.sh.j2
> >
> > The Bash script can then be converted to RST, e.g.:
> >
> >
> https://thirdparty.logs.rdoproject.org/jenkins-tripleo-quickstart-queens-rdo_trunk-baremetal-dell_fc430_envB-single_nic_vlans-27/docs/build/
> >
> > Source Code:
> >
> >
> https://github.com/openstack/tripleo-quickstart-extras/tree/master/roles/collect-logs
> >
> > I really liked this approach and while I don't want to sound like selling
> > other people's work, I'm wondering if there is still an interest among
> the
> > broader OpenStack community in automating documentation like this?
> >
> > Thanks,
> > pk
> >
>
> Weren't the folks doing the training-labs or training-guides taking a
> similar approach? IIRC, they ended up implementing what amounted to
> their own installer for OpenStack, and then ended up with all of the
> associated upgrade and testing burden.
>
> I like the idea of trying to use some automation from this, but I wonder
> if we'd be better off extracting data from other tools, rather than
> building a new one.
>
> Doug
>

So there really isn't anything new to create, the work is done and executed
on every tripleo change that runs in rdo-cloud.

Instead of dismissing the idea upfront I'm more inclined to set an
achievable small step to see how well it works.  My thought would be to
focus on the upcoming all-in-one installer and the automated doc generated
with that workflow.  I'd like to target publishing the all-in-one tripleo
installer doc to [1] for Stein and of course a section of tripleo.org.

What do you think?

[1] https://docs.openstack.org/queens/deploy/



>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [ci][infra][tripleo] Multi-staged check pipelines for Zuul v3 proposal

[Wesley Hayutin]

On Tue, May 15, 2018 at 11:42 AM Jeremy Stanley  wrote:

> On 2018-05-15 17:31:07 +0200 (+0200), Bogdan Dobrelya wrote:
> [...]
> > * upload into a swift container, with an automatic expiration set, the
> > de-duplicated and compressed tarball created with something like:
> >   # docker save $(docker images -q) | gzip -1 > all.tar.xz
> > (I expect it will be something like a 2G file)
> > * something similar for DLRN repos prolly, I'm not an expert for this
> part.
> >
> > Then those stored artifacts to be picked up by the next step in the
> graph,
> > deploying undercloud and overcloud in the single step, like:
> > * fetch the swift containers with repos and container images
> [...]
>
> I do worry a little about network fragility here, as well as
> extremely variable performance. Randomly-selected job nodes could be
> shuffling those files halfway across the globe so either upload or
> download (or both) will experience high round-trip latency as well
> as potentially constrained throughput, packet loss,
> disconnects/interruptions and so on... all the things we deal with
> when trying to rely on the Internet, except magnified by the
> quantity of data being transferred about.
>
> Ultimately still worth trying, I think, but just keep in mind it may
> introduce more issues than it solves.
> --
> Jeremy Stanley
>

Question...   If we were to build or update the containers that need an
update and I'm assuming the overcloud images here as well as a parent job.

The content would then sync to a swift file server on a central point for
ALL the openstack providers or it would be sync'd to each cloud?

Not to throw too much cold water on the idea, but...
I wonder if the time to upload and download the containers and images would
significantly reduce any advantage this process has.

Although centralizing the container updates and images on a per check job
basis sounds attractive, I get the sense we need to be very careful and
fully vett the idea.  At the moment it's also an optimization ( maybe ) so
I don't see this as a very high priority atm.

Let's bring the discussion the tripleo meeting next week.  Thanks all!



> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [ci][infra][tripleo] Multi-staged check pipelines for Zuul v3 proposal

[Wesley Hayutin]

On Mon, May 14, 2018 at 3:16 PM Sagi Shnaidman  wrote:

> Hi, Bogdan
>
> I like the idea with undercloud job. Actually if undercloud fails, I'd
> stop all other jobs, because it doens't make sense to run them. Seeing the
> same failure in 10 jobs doesn't add too much. So maybe adding undercloud
> job as dependency for all multinode jobs would be great idea. I think it's
> worth to check also how long it will delay jobs. Will all jobs wait until
> undercloud job is running? Or they will be aborted when undercloud job is
> failing?
>
> However I'm very sceptical about multinode containers and scenarios jobs,
> they could fail because of very different reasons, like race conditions in
> product or infra issues. Having skipping some of them will lead to more
> rechecks from devs trying to discover all problems in a row, which will
> delay the development process significantly.
>
> Thanks
>

I agree on both counts w/ Sagi here.
Thanks Sagi

>
>
> On Mon, May 14, 2018 at 7:15 PM, Bogdan Dobrelya 
> wrote:
>
>> An update for your review please folks
>>
>> Bogdan Dobrelya  writes:
>>>
>>> Hello.
 As Zuul documentation [0] explains, the names "check", "gate", and
 "post"  may be altered for more advanced pipelines. Is it doable to
 introduce, for particular openstack projects, multiple check
 stages/steps as check-1, check-2 and so on? And is it possible to make
 the consequent steps reusing environments from the previous steps
 finished with?

 Narrowing down to tripleo CI scope, the problem I'd want we to solve
 with this "virtual RFE", and using such multi-staged check pipelines,
 is reducing (ideally, de-duplicating) some of the common steps for
 existing CI jobs.

>>>
>>> What you're describing sounds more like a job graph within a pipeline.
>>> See:
>>> https://docs.openstack.org/infra/zuul/user/config.html#attr-job.dependencies
>>> for how to configure a job to run only after another job has completed.
>>> There is also a facility to pass data between such jobs.
>>>
>>> ... (skipped) ...
>>>
>>> Creating a job graph to have one job use the results of the previous job
>>> can make sense in a lot of cases.  It doesn't always save *time*
>>> however.
>>>
>>> It's worth noting that in OpenStack's Zuul, we have made an explicit
>>> choice not to have long-running integration jobs depend on shorter pep8
>>> or tox jobs, and that's because we value developer time more than CPU
>>> time.  We would rather run all of the tests and return all of the
>>> results so a developer can fix all of the errors as quickly as possible,
>>> rather than forcing an iterative workflow where they have to fix all the
>>> whitespace issues before the CI system will tell them which actual tests
>>> broke.
>>>
>>> -Jim
>>>
>>
>> I proposed a few zuul dependencies [0], [1] to tripleo CI pipelines for
>> undercloud deployments vs upgrades testing (and some more). Given that
>> those undercloud jobs have not so high fail rates though, I think Emilien
>> is right in his comments and those would buy us nothing.
>>
>> From the other side, what do you think folks of making the
>> tripleo-ci-centos-7-3nodes-multinode depend on
>> tripleo-ci-centos-7-containers-multinode [2]? The former seems quite faily
>> and long running, and is non-voting. It deploys (see featuresets configs
>> [3]*) a 3 nodes in HA fashion. And it seems almost never passing, when the
>> containers-multinode fails - see the CI stats page [4]. I've found only a 2
>> cases there for the otherwise situation, when containers-multinode fails,
>> but 3nodes-multinode passes. So cutting off those future failures via the
>> dependency added, *would* buy us something and allow other jobs to wait
>> less to commence, by a reasonable price of somewhat extended time of the
>> main zuul pipeline. I think it makes sense and that extended CI time will
>> not overhead the RDO CI execution times so much to become a problem. WDYT?
>>
>> [0] https://review.openstack.org/#/c/568275/
>> [1] https://review.openstack.org/#/c/568278/
>> [2] https://review.openstack.org/#/c/568326/
>> [3]
>> https://docs.openstack.org/tripleo-quickstart/latest/feature-configuration.html
>> [4] http://tripleo.org/cistatus.html
>>
>> * ignore the column 1, it's obsolete, all CI jobs now using configs
>> download AFAICT...
>>
>> --
>> Best regards,
>> Bogdan Dobrelya,
>> Irc #bogdando
>>
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>
>
>
> --
> Best regards
> Sagi Shnaidman
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailm

Re: [openstack-dev] [ci][infra][tripleo] Multi-staged check pipelines for Zuul v3 proposal

[Wesley Hayutin]

On Tue, May 15, 2018 at 1:29 PM James E. Blair  wrote:

> Jeremy Stanley  writes:
>
> > On 2018-05-15 09:40:28 -0700 (-0700), James E. Blair wrote:
> > [...]
> >> We're also talking about making a new kind of job which can continue to
> >> run after it's "finished" so that you could use it to do something like
> >> host a container registry that's used by other jobs running on the
> >> change.  We don't have that feature yet, but if we did, would you prefer
> >> to use that instead of the intermediate swift storage?
> >
> > If the subsequent jobs depending on that one get nodes allocated
> > from the same provider, that could solve a lot of the potential
> > network performance risks as well.
>
> That's... tricky.  We're *also* looking at affinity for buildsets, and
> I'm optimistic we'll end up with something there eventually, but that's
> likely to be a more substantive change and probably won't happen as
> soon.  I do agree it will be nice, especially for use cases like this.
>
> -Jim
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


There is a lot here to unpack and discuss, but I really like the ideas I'm
seeing.
Nice work Bogdan!  I've added it the tripleo meeting agenda for next week
so we can continue socializing the idea and get feedback.

Thanks!

https://etherpad.openstack.org/p/tripleo-meeting-items
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo] Zuul repo insertion in update/upgrade CI

[Wesley Hayutin]

On Mon, May 14, 2018 at 11:36 AM Jiří Stránský  wrote:

> Hi,
>
> this is mainly for CI folks and whom-it-may-concern.
>
> Recently we came across the topic of how to enable/disable zuul repos at
> various places in the CI jobs. For normal deploy jobs there's no need to
> customize, but for update/upgrade jobs there is. It's not entirely
> straightforward and there's quite a variety of enable/disable spots and
> combinations which can be useful.
>
> Even though improvements in this area are not very likely to get
> implemented right away, i had some thoughts on the topic so i wanted to
> capture them. I put the ideas into an etherpad:
>
> https://etherpad.openstack.org/p/tripleo-ci-zuul-repo-insertion
>
> Feel free to put some more thoughts there or ping me on IRC with
> anything related.
>
>
> Thanks
>
> Jirka
>
>
Thanks Jirka!!


> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo] tripleo upstream gate outtage, was: -> gate jobs impacted RAX yum mirror

[Wesley Hayutin]

On Sun, May 13, 2018 at 11:50 PM Tristan Cacqueray 
wrote:

> On May 14, 2018 2:44 am, Wesley Hayutin wrote:
> [snip]
> > I do think it would be helpful to say have a one week change window where
> > folks are given the opportunity to preflight check a new image and the
> > potential impact on the job workflow the updated image may have.
> [snip]
>
> How about adding a periodic job that setup centos-release-cr in a pre
> task? This should highlight issues with up-coming updates:
> https://wiki.centos.org/AdditionalResources/Repositories/CR
>
> -Tristan
>

Thanks for the suggestion Tristan, going to propose using this repo at the
next TripleO mtg.

Thanks


> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo] tripleo upstream gate outtage, was: -> gate jobs impacted RAX yum mirror

[Wesley Hayutin]

On Mon, May 14, 2018 at 12:37 PM Jeremy Stanley  wrote:

> On 2018-05-14 09:57:17 -0600 (-0600), Wesley Hayutin wrote:
> > On Mon, May 14, 2018 at 10:36 AM Jeremy Stanley 
> wrote:
> [...]
> > > Couldn't a significant burst of new packages cause the same
> > > symptoms even without it being tied to a minor version increase?
> >
> > Yes, certainly this could happen outside of a minor update of the
> > baseos.
>
> Thanks for confirming. So this is not specifically a CentOS minor
> version increase issue, it's just more likely to occur at minor
> version boundaries.
>

Correct, you got it


>
> > So the only thing out of our control is the package set on the
> > base nodepool image. If that suddenly gets updated with too many
> > packages, then we have to scramble to ensure the images and
> > containers are also udpated.
>
> It's still unclear to me why the packages on the test instance image
> (i.e. the "container host") are related to the packages in the
> container guest images at all. That would seem to be the whole point
> of having containers?
>

You are right, just note some services are not 100% containerized yet.
This doesn't happen overnight it's a process and we're getting there.


>
> > If there is a breaking change in the nodepool image for example
> > [a], we have to react to and fix that as well.
>
> I would argue that one is a terrible workaround which happened to
> show its warts. We should fix DIB's pip-and-virtualenv element
> rather than continue rely on side effects of pinning RPM versions.
> I've commented to that effect on https://launchpad.net/bugs/1770298
> just now.
>
>
k.. thanks


> > > It sounds like a problem with how the jobs are designed
> > > and expectations around distros slowly trickling package updates
> > > into the series without occasional larger bursts of package deltas.
> > > I'd like to understand more about why you upgrade packages inside
> > > your externally-produced container images at job runtime at all,
> > > rather than relying on the package versions baked into them.
> >
> > We do that to ensure the gerrit review itself and it's
> > dependencies are built via rpm and injected into the build. If we
> > did not do this the job would not be testing the change at all.
> > This is a result of being a package based deployment for better or
> > worse.
> [...]
>
> Now I'll risk jumping to proposing solutions, but have you
> considered building those particular packages in containers too?
> That way they're built against the same package versions as will be
> present in the other container images you're using rather than to
> the package versions on the host, right? Seems like it would
> completely sidestep the problem.
>

So a little background.  The containers and images used in TripleO are
rebuilt multiple times each day via periodic jobs, when they pass our
criteria they are pushed out and used upstream.
Each zuul change and it's dependencies can potentially impact a few or all
the containers in play.   We can not rebuild all the containers due to time
constraints in each job.  We have been able to mount and yum update the
containers involved with the zuul change.

Latest patch to fine tune that process is here
https://review.openstack.org/#/c/567550/


>
> > An enhancement could be to stage the new images for say one week
> > or so. Do we need the CentOS updates immediately? Is there a
> > possible path that does not create a lot of work for infra, but
> > also provides some space for projects to prep for the consumption
> > of the updates?
> [...]
>
> Nodepool builds new images constantly, but at least daily. Part of
> this is to prevent the delta of available packages/indices and other
> files baked into those images from being more than a day or so stale
> at any given point in time. The older the image, the more packages
> (on average) jobs will need to download if they want to test with
> latest package versions and the more strain it will put on our
> mirrors and on our bandwidth quotas/donors' networks.
>

Sure that makes perfect sense.  We do the same with our containers and
images.


>
> There's also a question of retention, if we're building images at
> least daily but keeping them around for 7 days (storage on the
> builders, tenant quotas for Glance in our providers) as well as the
> explosion of additional nodes we'd need since we pre-boot nodes with
> each of our images (and the idea as I understand it is that you
> would want jobs to be able to select between any of them). One
> option, I suppose, woul

Re: [openstack-dev] [tripleo] tripleo upstream gate outtage, was: -> gate jobs impacted RAX yum mirror

[Wesley Hayutin]

On Mon, May 14, 2018 at 12:08 PM Clark Boylan  wrote:

> On Mon, May 14, 2018, at 8:57 AM, Wesley Hayutin wrote:
> > On Mon, May 14, 2018 at 10:36 AM Jeremy Stanley 
> wrote:
> >
> > > On 2018-05-14 07:07:03 -0600 (-0600), Wesley Hayutin wrote:
> > > [...]
>
> snip
>
> > >
> > > This _doesn't_ sound to me like a problem with how we've designed
> > > our infrastructure, unless there are additional details you're
> > > omitting.
> >
> >
> > So the only thing out of our control is the package set on the base
> > nodepool image.
> > If that suddenly gets updated with too many packages, then we have to
> > scramble to ensure the images and containers are also udpated.
> > If there is a breaking change in the nodepool image for example [a], we
> > have to react to and fix that as well.
>
> Aren't the container images independent of the hosting platform (eg what
> infra hosts)? I'm not sure I understand why the host platform updating
> implies all the container images must also be updated.
>

You make a fine point here, I think as with anything there are some bits
that are still being worked on. At this moment it's my understanding that
pacemaker and possibly a few others components are not 100% containerized
atm.  I'm not an expert in the subject and my understanding may not be
correct.  Untill you are 100% containerized there may still be some
dependencies on the base image and an impact from changes.


>
> >
> >
> > > It sounds like a problem with how the jobs are designed
> > > and expectations around distros slowly trickling package updates
> > > into the series without occasional larger bursts of package deltas.
> > > I'd like to understand more about why you upgrade packages inside
> > > your externally-produced container images at job runtime at all,
> > > rather than relying on the package versions baked into them.
> >
> >
> > We do that to ensure the gerrit review itself and it's dependencies are
> > built via rpm and injected into the build.
> > If we did not do this the job would not be testing the change at all.
> >  This is a result of being a package based deployment for better or
> worse.
>
> You'd only need to do that for the change in review, not the entire system
> right?
>

Correct there is no intention of updating the entire distribution in run
time, the intent is to have as much updated in our jobs that build the
containers and images.
Only the rpm built zuul change should be included in the update, however
some zuul changes require a CentOS base package that was not previously
installed on the container e.g. a new python dependency introduced in a
zuul change.  Previously we had not enabled any CentOS repos in the
container update, but found that was not viable 100% of the time.

We have a change to further limit the scope of the update which should help
[1], especialy when facing a minor version update.

 [1] https://review.openstack.org/#/c/567550/

>
> >
>
> snip
>
> > > Our automation doesn't know that there's a difference between
> > > packages which were part of CentOS 7.4 and 7.5 any more than it
> > > knows that there's a difference between Ubuntu 16.04.2 and 16.04.3.
> > > Even if we somehow managed to pause our CentOS image updates
> > > immediately prior to 7.5, jobs would still try to upgrade those
> > > 7.4-based images to the 7.5 packages in our mirror, right?
> > >
> >
> > Understood, I suspect this will become a more widespread issue as
> > more projects start to use containers ( not sure ).  It's my
> understanding
> > that
> > there are some mechanisms in place to pin packages in the centos nodepool
> > image so
> > there has been some thoughts generally in the area of this issue.
>
> Again, I think we need to understand why containers would make this worse
> not better. Seems like the big feature everyone talks about when it comes
> to containers is isolating packaging whether that be python packages so
> that nova and glance can use a different version of oslo or cohabitating
> software that would otherwise conflict. Why do the packages on the host
> platform so strongly impact your container package lists?
>

I'll let others comment on that, however my thought is you don't move from
A -> Z in one step and containers do not make everything easier
immediately.  Like most things, it takes a little time.

>
> >
> > TripleO may be the exception to the rule here and that is fine, I'm more
> > interested in exploring
> > the possibilities of delivering

Re: [openstack-dev] [tripleo] tripleo upstream gate outtage, was: -> gate jobs impacted RAX yum mirror

[Wesley Hayutin]

On Mon, May 14, 2018 at 10:36 AM Jeremy Stanley  wrote:

> On 2018-05-14 07:07:03 -0600 (-0600), Wesley Hayutin wrote:
> [...]
> > I think you may be conflating the notion that ubuntu or rhel/cent
> > can be updated w/o any issues to applications that run atop of the
> > distributions with what it means to introduce a minor update into
> > the upstream openstack ci workflow.
> >
> > If jobs could execute w/o a timeout the tripleo jobs would have
> > not gone red.  Since we do have constraints in the upstream like a
> > timeouts and others we have to prepare containers, images etc to
> > work efficiently in the upstream.  For example, if our jobs had
> > the time to yum update the roughly 120 containers in play in each
> > job the tripleo jobs would have just worked.  I am not advocating
> > for not having timeouts or constraints on jobs, however I am
> > saying this is an infra issue, not a distribution or distribution
> > support issue.
> >
> > I think this is an important point to consider and I view it as
> > mostly unrelated to the support claims by the distribution.  Does
> > that make sense?
> [...]
>
> Thanks, the thread jumped straight to suggesting costly fixes
> (separate images for each CentOS point release, adding an evaluation
> period or acceptance testing for new point releases, et cetera)
> without coming anywhere close to exploring the problem space. Is
> your only concern that when your jobs started using CentOS 7.5
> instead of 7.4 they took longer to run?


Yes, If they had unlimited time to run, our workflow would have everything
updated to CentOS 7.5 in the job itself and I would expect everything to
just work.


> What was the root cause? Are
> you saying your jobs consume externally-produced artifacts which lag
> behind CentOS package updates?


Yes, TripleO has externally produced overcloud images, and containers both
of which can be yum updated but we try to ensure they are frequently
recreated so the yum transaction is small.


> Couldn't a significant burst of new
> packages cause the same symptoms even without it being tied to a
> minor version increase?
>

Yes, certainly this could happen outside of a minor update of the baseos.


>
> This _doesn't_ sound to me like a problem with how we've designed
> our infrastructure, unless there are additional details you're
> omitting.


So the only thing out of our control is the package set on the base
nodepool image.
If that suddenly gets updated with too many packages, then we have to
scramble to ensure the images and containers are also udpated.
If there is a breaking change in the nodepool image for example [a], we
have to react to and fix that as well.


> It sounds like a problem with how the jobs are designed
> and expectations around distros slowly trickling package updates
> into the series without occasional larger bursts of package deltas.
> I'd like to understand more about why you upgrade packages inside
> your externally-produced container images at job runtime at all,
> rather than relying on the package versions baked into them.


We do that to ensure the gerrit review itself and it's dependencies are
built via rpm and injected into the build.
If we did not do this the job would not be testing the change at all.
 This is a result of being a package based deployment for better or worse.


> It
> seems like you're arguing that the existence of lots of new package
> versions which aren't already in your container images is the
> problem, in which case I have trouble with the rationalization of it
> being "an infra issue" insofar as it requires changes to the
> services as provided by the OpenStack Infra team.
>
> Just to be clear, we didn't "introduce a minor update into the
> upstream openstack ci workflow." We continuously pull CentOS 7
> packages into our package mirrors, and continuously rebuild our
> centos-7 images from whatever packages the distro says are current.
>

Understood, which I think is fine and probably works for most projects.
An enhancement could be to stage the new images for say one week or so.
Do we need the CentOS updates immediately? Is there a possible path that
does not create a lot of work for infra, but also provides some space for
projects
to prep for the consumption of the updates?


> Our automation doesn't know that there's a difference between
> packages which were part of CentOS 7.4 and 7.5 any more than it
> knows that there's a difference between Ubuntu 16.04.2 and 16.04.3.
> Even if we somehow managed to pause our CentOS image updates
> immediately prior to 7.5, jobs would still try to upgrade those
> 7.4-based images to the 7.5 packages in our mirror, right?
>

U

Re: [openstack-dev] [tripleo] tripleo upstream gate outtage, was: -> gate jobs impacted RAX yum mirror

[Wesley Hayutin]

On Sun, May 13, 2018 at 11:30 PM Jeremy Stanley  wrote:

> On 2018-05-13 20:44:25 -0600 (-0600), Wesley Hayutin wrote:
> [...]
> > I do think it would be helpful to say have a one week change
> > window where folks are given the opportunity to preflight check a
> > new image and the potential impact on the job workflow the updated
> > image may have. If I could update or create a non-voting job w/
> > the new image that would provide two things.
> >
> > 1. The first is the head's up, this new minor version of centos is
> > coming into the system and you have $x days to deal with it.
> >
> > 2. The ability to build a few non-voting jobs w/ the new image to
> > see what kind of impact it has on the workflow and deployments.
> [...]
>
> While I can see where you're coming from, right now even the Infra
> team doesn't know immediately when a new CentOS minor release starts
> to be used. The packages show up in the mirrors automatically and
> images begin to be built with them right away. There isn't a
> conscious "switch" which is thrown by anyone. This is essentially
> the same way we treat Ubuntu LTS point releases as well. If this is
> _not_ the way RHEL/CentOS are intended to be consumed (i.e. just
> upgrade to and run the latest packages available for a given major
> release series) then we should perhaps take a step back and
> reevaluate this model.


I think you may be conflating the notion that ubuntu or rhel/cent can be
updated w/o any issues to applications that run atop of the distributions
with what it means to introduce a minor update into the upstream openstack
ci workflow.

If jobs could execute w/o a timeout the tripleo jobs would have not gone
red.  Since we do have constraints in the upstream like a timeouts and
others we have to prepare containers, images etc to work efficiently in the
upstream.  For example, if our jobs had the time to yum update the roughly
120 containers in play in each job the tripleo jobs would have just
worked.  I am not advocating for not having timeouts or constraints on
jobs, however I am saying this is an infra issue, not a distribution or
distribution support issue.

I think this is an important point to consider and I view it as mostly
unrelated to the support claims by the distribution.  Does that make sense?
Thanks




> For now we have some fairly deep-driven
> assumptions in that regard which are reflected in the Linux
> distributions support policy of our project testing interface as
> documented in OpenStack governance.
> --
> Jeremy Stanley
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo] tripleo upstream gate outtage, was: -> gate jobs impacted RAX yum mirror

[Wesley Hayutin]

On Sun, May 13, 2018 at 11:25 AM Jeremy Stanley  wrote:

> On 2018-05-13 08:25:25 -0600 (-0600), Wesley Hayutin wrote:
> [...]
> > We need to in coordination with the infra team be able to pin / lock
> > content for production check and gate jobs while also have the ability to
> > stage new content e.g. centos 7.5 with experimental or periodic jobs.
> [...]
>
> It looks like adjustments would be needed to DIB's centos-minimal
> element if we want to be able to pin it to specific minor releases.
> However, having to rotate out images in the fashion described would
> be a fair amount of manual effort and seems like it would violate
> our support expectations in governance if we end up pinning to older
> minor versions (for major LTS versions on the other hand, we expect
> to undergo this level of coordination but they come at a much slower
> pace with a lot more advance warning). If we need to add controlled
> roll-out of CentOS minor version updates, this is really no better
> than Fedora from the Infra team's perspective and we've already said
> we can't make stable branch testing guarantees for Fedora due to the
> complexity involved in using different releases for each branch and
> the need to support our stable branches longer than the distros are
> supporting the releases on which we're testing.
>

This is good insight Jeremy, thanks for replying.



>
> For example, how long would the distro maintainers have committed to
> supporting RHEL 7.4 after 7.5 was released? Longer than we're
> committing to extended maintenance on our stable/queens branches? Or
> would you expect projects to still continue to backport support for
> these minor platform bumps to all their stable branches too? And
> what sort of grace period should we give them before we take away
> the old versions? Also, how many minor versions of CentOS should we
> expect to end up maintaining in parallel? (Remember, every
> additional image means that much extra time to build and upload to
> all our providers, as well as that much more storage on our builders
> and in our Glance quotas.)
> --
> Jeremy Stanley
>

I think you may be describing a level of support that is far greater than
what I was thinking. I also don't want to tax the infra team w/ n+ versions
of the baseos to support.
I do think it would be helpful to say have a one week change window where
folks are given the opportunity to preflight check a new image and the
potential impact on the job workflow the updated image may have.  If I
could update or create a non-voting job w/ the new image that would provide
two things.

1. The first is the head's up, this new minor version of centos is coming
into the system and you have $x days to deal with it.
2. The ability to build a few non-voting jobs w/ the new image to see what
kind of impact it has on the workflow and deployments.

In this case the updated 7.5 CentOS image worked fine w/ TripleO, however
it did cause our gates to go red because..
a. when we update containers w/ zuul dependendencies all the base-os
updates were pulled in and jobs timed out.
b. a kernel bug workaround with virt-customize failed to work due the
kernel packages changed ( 3rd party job )
c. the containers we use were not yet at CentOS 7.5 but the bm image was
causing issues w/ pacemaker.
d. there may be a few more that I am forgetting, but hopefully the point is
made.

We can fix a lot of the issues and I'm not blaming anyone because if we
(tripleo ) thought of all the corner cases with our workflow we would have
been able to avoid some of these issues.  However it does seem like we get
hit by $something every time we update a minor version of the baseos.  My
preference would be to have a heads up and work through the issues than to
go immediately red and unable to merge patches.  I don't know if other
teams get impacted in similiar ways, and I understand this is a big ship
and updating CentOS may work just fine for everyone else.

Thanks all for your time and effort!




> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo] tripleo upstream gate outtage, was: -> gate jobs impacted RAX yum mirror

[Wesley Hayutin]

On Sat, May 12, 2018 at 11:45 PM Emilien Macchi  wrote:

> On Sat, May 12, 2018 at 9:10 AM, Wesley Hayutin 
> wrote:
>>
>> 2. Shortly after #1 was resolved CentOS released 7.5 which comes directly
>> into the upstream repos untested and ungated.  Additionally the associated
>> qcow2 image and container-base images were not updated at the same time as
>> the yum repos.  https://bugs.launchpad.net/tripleo/+bug/1770355
>>
>
> Why do we have this situation everytime the OS is upgraded to a major
> version? Can't we test the image before actually using it? We could have
> experimental jobs testing latest image and pin gate images to a specific
> one?
> Like we could configure infra to deploy centos 7.4 in our gate and 7.5 in
> experimental, so we can take our time to fix eventual problems and make the
> switch when we're ready, instead of dealing with fires (that usually come
> all together).
>
> It would be great to make a retrospective on this thing between tripleo ci
> & infra folks, and see how we can improve things.
>

I agree,
We need to in coordination with the infra team be able to pin / lock
content for production check and gate jobs while also have the ability to
stage new content e.g. centos 7.5 with experimental or periodic jobs.
In this particular case the ci team did check the tripleo deployment w/
centos 7.5 updates, however we did not stage or test what impact the centos
minor update would have on the upstream job workflow.
The key issue is that the base centos image used upstream can not be pinned
by the ci team, if say we could pin that image the ci team could pin the
centos repos used in ci and run staging jobs on the latest centos content.

I'm glad that you also see the need for some amount of coordination here,
I've been in contact with a few folks to initiate the conversation.

In an unrelated note, Sagi and I just fixed the network latency issue on
our promotion server, it was related to DNS.  Automatic promotions should
be back online.
Thanks all.


> --
> Emilien Macchi
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [tripleo] tripleo upstream gate outtage, was: -> gate jobs impacted RAX yum mirror

[Wesley Hayutin]

On Wed, May 9, 2018 at 10:43 PM Wesley Hayutin  wrote:

> FYI.. https://bugs.launchpad.net/tripleo/+bug/1770298
>
> I'm on #openstack-infra chatting w/ Ian atm.
> Thanks
>
>
Greetings,

I wanted to update  everyone on the status of the upstream tripleo check
and gate jobs.
There have been a series of infra related issues that caused the upstream
tripleo gates to go red.

1. The first issue hit was
https://bugs.launchpad.net/tripleo/+bug/1770298 which
caused package install errors
2. Shortly after #1 was resolved CentOS released 7.5 which comes directly
into the upstream repos untested and ungated.  Additionally the associated
qcow2 image and container-base images were not updated at the same time as
the yum repos.  https://bugs.launchpad.net/tripleo/+bug/1770355
3.  Related to #2 the container and bm image rpms were not in sync causing
https://bugs.launchpad.net/tripleo/+bug/1770692
4. Building the bm images was failing due to an open issue with the centos
kernel, thanks to Yatin and Alfredo for
https://review.rdoproject.org/r/#/c/13737/
5. To ensure the containers are updated to the latest rpms at build time,
we have the following patch from Alex
https://review.openstack.org/#/c/567636/.
6.  I also noticed that we are building the centos-base container in our
container build jobs, however it is not pushed out to the container
registeries because it is not included in the tripleo-common repo
<https://github.com/openstack/tripleo-common/blob/master/container-images/overcloud_containers.yaml.j2>
I would like to discuss this with some of the folks working on containers.
If we had an updated centos-base container I think some of these issues
would have been prevented.

The above issues were resolved, and the master promotion jobs all had
passed.  Thanks to all who were involved!

Once the promotion jobs pass and report status to the dlrn_api, a promotion
was triggered automatically to upload the promoted images, containers, and
updated dlrn hash.  This failed due to network latency in the tenant where
the tripleo-ci infra is hosted.  The issue is tracked here
https://bugs.launchpad.net/tripleo/+bug/1770860

Matt Young and myself worked well into the evening on Friday to diagnose
the issue and ended up having to execute the image, container and dlrn_hash
promotion outside of our tripleo-infra tenant.  Thanks to Matt for his
effort.

At the moment I have updated the ci status in #tripleo, the master check
and gate jobs are green in the upstream which should unblock merging most
patches.  The status of stable branches and third party ci is still being
investigated.

Automatic promotions are blocked until the network issues in the
tripleo-infra tenant are resolved.  The bug is marked with alert in
#tripleo.  Please see #tripleo for future status updates.

Thanks all
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [tripleo] gate jobs impacted RAX yum mirror

[Wesley Hayutin]

FYI.. https://bugs.launchpad.net/tripleo/+bug/1770298

I'm on #openstack-infra chatting w/ Ian atm.
Thanks
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo] Migration to Storyboard

[Wesley Hayutin]

On Wed, May 9, 2018 at 3:25 PM Alex Schultz  wrote:

> Hello tripleo folks,
>
> So we've been experimenting with migrating some squads over to
> storyboard[0] but this seems to be causing more issues than perhaps
> it's worth.  Since the upstream community would like to standardize on
> Storyboard at some point, I would propose that we do a cut over of all
> the tripleo bugs/blueprints from Launchpad to Storyboard.
>
> In the irc meeting this week[1], I asked that the tripleo-ci team make
> sure the existing scripts that we use to monitor bugs for CI support
> Storyboard.  I would consider this a prerequisite for the migration.
> I am thinking it would be beneficial to get this done before or as
> close to M2.
>
> Thoughts, concerns, etc?
>

Just clarifying.  You would like to have the tooling updated by M2, which
is fine I think.  However squads are not expected to change all their
existing procedures by M2 correct?   I'm concerned about migrating our
current kanban boards to storyboard by M2.

Thanks


>
> Thanks,
> -Alex
>
> [0] https://storyboard.openstack.org/#!/project_group/76
> [1]
> http://eavesdrop.openstack.org/meetings/tripleo/2018/tripleo.2018-05-08-14.00.log.html#l-42
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Overriding project-templates in Zuul

[Wesley Hayutin]

On Tue, May 1, 2018 at 1:23 PM Emilien Macchi  wrote:

> On Tue, May 1, 2018 at 10:02 AM, James E. Blair 
> wrote:
> [...]
>
> Okay, let's summarize:
>>
>> Proposal 1: All project-template and project-local job variants matching
>> the item's branch must also match the item.
>>
>> * Files and irrelevant-files on project-template and project stanzas are
>>   essentially combined in a set intersection.
>> * It's possible to further reduce the scope of jobs, but not expand.
>> * Files and irrelevant-files are still independent matchers, and if both
>>   are present, both must match.
>> * It's not possible to alter a job attribute by adding a project-local
>>   variant with only a files matcher (it would cause the whole job to run
>>   or not run).  But it's still possible to do that in the main job
>>   definition itself.
>>
>> Proposal 2: Files and irrelevant-files are treated as overwriteable
>> attributes and evaluated after branch-matching variants are combined.
>>
>> * Files and irrelevant-files are overwritten, so the last value
>>   encountered when combining all the matching variants (looking only at
>>   branches) wins.
>> * Files and irrelevant-files will be treated as a pair, so that if
>>   "irrelevant-files" appears, it will erase a previous "files"
>>   attribute.
>> * It's possible to both reduce and expand the scope of jobs, but the
>>   user may need to manually copy values from a parent or other variant
>>   in order to do so.
>> * It will no longer be possible to alter a job attribute by adding a
>>   variant with only a files matcher -- in all cases files and
>>   irrelevant-files are used solely to determine whether the job is run,
>>   not to determine whether to apply a variant.
>>
>> I think both would be good solutions to the problem.  The key points for
>> me are whether we want to keep the "alter a job attribute with variant
>> with a files matcher" functionality (the "rebuild_index" example from
>> above), and whether the additional control of overwriting the matchers
>> (at the cost of redundancy in configuration) is preferable to combining
>> the matchers.
>>
>
> In the case of TripleO, I think proposal 2 is what we want.
> We have stanzas defined in the templates definitions in
> openstack-infra/tripleo-ci repo, but really want to override the file rules
> per repo (openstack/tripleo-quickstart for example) and I don't think we
> want to have them both matching but so the last value encountered would win.
> I'll let TripleO CI squad to give more thoughts though.
>
> Thanks,
> --
> Emilien Macchi
>

I agree,
Proposal #2 makes the most sense to me and seems more straight forward in
that you have the ability to override and that the project overriding would
need to handle both files and irrelevant-files from scratch.

Nice write up



> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo] The Weekly Owl - 17th Edition

[Wesley Hayutin]

On Tue, Apr 17, 2018 at 9:44 PM Emilien Macchi  wrote:

> Note: this is the seventeeth edition of a weekly update of what happens in
> TripleO.
> The goal is to provide a short reading (less than 5 minutes) to learn
> where we are and what we're doing.
> Any contributions and feedback are welcome.
> Link to the previous version:
> http://lists.openstack.org/pipermail/openstack-dev/2018-April/129255.html
>
> +-+
> | General announcements |
> +-+
>
> +--> Rocky milestone 1 will be released this week (probably tomorrow)!
> +--> (reminder) if you're looking at reproducing a CI job, checkout:
> https://docs.openstack.org/tripleo-docs/latest/contributor/reproduce-ci.html
>
> +--+
> | Continuous Integration |
> +--+
>
> +--> Ruck is quiquell and Rover is panda. Please let them know any new CI
> issue.
> +--> Master promotion is 1 day, Queens is 2 days, Pike is 4 days and Ocata
> is 5 days.
> +--> Efforts around libvirt based multinode reproducer, see
> https://trello.com/c/JEGLSVh6/323-reproduce-ci-jobs-with-libvirt
> +--> More: https://etherpad.openstack.org/p/tripleo-ci-squad-meeting and
> https://goo.gl/D4WuBP
>

So, just to add some context.  We would like to be able to setup libvirt
guests in the same way nodepool nodes are setup to allow the ci team and
others to reexecute upstream ci jobs on libvirt using the exact workflow
that upstream jobs take.

A reminder the current reproduce scripts are documented here [1].  We plan
on updating the current doc with our libvirt work when it is ready.
 Thanks all

[1] http://tripleo.org/contributor/reproduce-ci.html


>
>
> +-+
> | Upgrades |
> +-+
>
> +--> Progress on FFU CLI in tripleoclient, need reviews.
> +--> Work for containerized undercloud upgrades has been merged. Testing
> will make progress after rocky-m1 (with new tags).
> +--> More: https://etherpad.openstack.org/p/tripleo-upgrade-squad-status
>
> +---+
> | Containers |
> +---+
>
> +--> Still working on UX problems
> +--> Still working on container workflow, good progress last week where
> container prepare isn't needed. Now working on container updates.
> +--> Investigating how to bootstrap Docker + Registry before deploying
> containers
> +--> Progress on routed networks support
> +--> More: https://etherpad.openstack.org/p/tripleo-containers-sq
> uad-status
>
> +--+
> | config-download |
> +--+
>
> +--> Moving to config-download by default is coming very soon (once Ceph
> patches land).
> +--> Ceph was migrated and all patches are going to merge this week.
> +--> octavia/skydive migration is wip.
> +--> Improving deploy-steps-tasks.j2 to improve playbook readability and
> memory consumption
> +--> UI work is work in progress.
> +--> More: https://etherpad.openstack.org/p/tripleo-config-downlo
> ad-squad-status
>
> +--+
> | Integration |
> +--+
>
> +--> No updates.
> +--> More: https://etherpad.openstack.org/p/tripleo-integration-s
> quad-status
>
> +-+
> | UI/CLI |
> +-+
>
> +--> Efforts on config-download integration
> +--> Added type to ansible-playbook messages (feedback needed)
> +--> More: https://etherpad.openstack.org/p/tripleo-ui-cli-squad-status
>
> +---+
> | Validations |
> +---+
>
> +--> No updates.
> +--> More: https://etherpad.openstack.org/p/tripleo-validations-s
> quad-status
>
> +---+
> | Networking |
> +---+
>
> +--> No updates this week.
> +--> More: https://etherpad.openstack.org/p/tripleo-networking-sq
> uad-status
>
> +--+
> | Workflows |
> +--+
>
> +--> Need reviews, see etherpad.
> +--> Working on workflows v2
> +--> More: https://etherpad.openstack.org/p/tripleo-workflows-squad-status
>
> +---+
> | Security |
> +---+
>
> +--> Tomorrow's meeting is about Storyboard migration and Secret
> management.
> +--> More: https://etherpad.openstack.org/p/tripleo-security-squad
>
> ++
> | Owl fact  |
> ++
>
> Did you know owls were watching you while working on TripleO?
> Check this out:
> https://www.reddit.com/r/pics/comments/8cz8v0/owls_born_outside_of_office_window_wont_stop/
> (Thanks Wes for the link)
>
>
> Thanks all for reading and stay tuned!
> --
> Your fellow reporter, Emilien Macchi
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/op

Re: [openstack-dev] [tripleo] roadmap on containers workflow

[Wesley Hayutin]

On Tue, 10 Apr 2018 at 20:51 Emilien Macchi  wrote:

> Greetings,
>
> Steve Baker and I had a quick chat today about the work that is being done
> around containers workflow in Rocky cycle.
>
> If you're not familiar with the topic, I suggest to first read the
> blueprint to understand the context here:
> https://blueprints.launchpad.net/tripleo/+spec/container-prepare-workflow
>
> One of the great outcomes of this blueprint is that in Rocky, the operator
> won't have to run all the "openstack overcloud container" commands to
> prepare the container registry and upload the containers. Indeed, it'll be
> driven by Heat and Mistral mostly.
>
> But today our discussion extended on 2 uses-cases that we're going to
> explore and find how we can address them:
> 1) I'm a developer and want to deploy a containerized undercloud with
> customized containers (more or less related to the all-in-one discussions
> on another thread [1]).
> 2) I'm submitting a patch in tripleo-common (let's say a workflow) and
> need my patch to be tested when the undercloud is containerized (see [2]
> for an excellent example).
>
> Both cases would require additional things:
> - The container registry needs to be deployed *before* actually installing
> the undercloud.
> - We need a tool to update containers from this registry and *before*
> deploying them. We already have this tool in place in our CI for the
> overcloud (see [3] and [4]). Now we need a similar thing for the undercloud.
>
> Next steps:
> - Agree that we need to deploy the container-registry before the
> undercloud.
> - If agreed, we'll create a new Ansible role called
> ansible-role-container-registry that for now will deploy exactly what we
> have in TripleO, without extra feature.
> - Drive the playbook runtime from tripleoclient to bootstrap the container
> registry (which of course could be disabled in undercloud.conf).
> - Create another Ansible role that would re-use container-check tool but
> the idea is to provide a role to modify containers when needed, and we
> could also control it from tripleoclient. The role would be using
> the ContainerImagePrepare parameter, which Steve is working on right now.
>

This all looks really good Emilien, thanks for sending it out.
Regarding the update of containers, we would just want to be 100% sure that
we can control which yum repositories are in play for the update.  Maybe it
will be done by the user prior to running the command, or maybe with some
flags to what ever command Steve is working on.
FYI.. we've noticed in CI that when the base os updates ( not baseos) are
included you tend to fail on at least on package download on one of the 50+
containers due to infra/network.  In CI we only enable baseos, dlrn updates
and the dependency change [1]

Thanks

[1]
https://github.com/openstack/tripleo-quickstart-extras/blob/master/roles/overcloud-prep-containers/templates/overcloud-prep-containers.sh.j2#L104-L109


>
> Feedback is welcome, thanks.
>
> [1] All-In-One thread:
> http://lists.openstack.org/pipermail/openstack-dev/2018-March/128900.html
> [2] Bug report when undercloud is containeirzed
> https://bugs.launchpad.net/tripleo/+bug/1762422
> [3] Tool to update containers if needed:
> https://github.com/imain/container-check
> [4] Container-check running in TripleO CI:
> https://review.openstack.org/#/c/558885/ and
> https://review.openstack.org/#/c/529399/
> --
> Emilien Macchi
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo] The Weekly Owl - 16th Edition

[Wesley Hayutin]

On Tue, 10 Apr 2018 at 19:24 Emilien Macchi  wrote:

> Note: this is the sixteenth edition of a weekly update of what happens in
> TripleO.
> The goal is to provide a short reading (less than 5 minutes) to learn
> where we are and what we're doing.
> Any contributions and feedback are welcome.
> Link to the previous version:
> http://lists.openstack.org/pipermail/openstack-dev/2018-April/129035.html
>
> +-+
> | General announcements |
> +-+
>
> +--> Rocky milestone 1 is next week! Please update your blueprints status
> accordingly)
>
> +--+
> | Continuous Integration |
> +--+
>
> +--> Rover is Arx and Ruck is Rafael. Please let them know any new CI
> issue.
> +--> Master promotion is 5 days, Queens is 12 days, Pike is 17 days and
> Ocata is 17 days.
>

FYI.. Queens promoted today
Master promotion is 5 days, Queens is 0 days, Pike is 17 days and Ocata is
17

Thanks Emilien!


> +--> Efforts around a simple "keystone-only" CI job across all branches.
> +--> Good progress on running Tempest for undercloud jobs, also tempest
> containerization.
> +--> More: https://etherpad.openstack.org/p/tripleo-ci-squad-meeting and
> https://goo.gl/D4WuBP
>
> +-+
> | Upgrades |
> +-+
>
> +--> Progress on FFU CLI in tripleoclient and FFU/Ceph as well.
> +--> Work on CI jobs for undercloud upgrades and containerized undercloud
> upgrades.
> +--> Need reviews, see etherpad
> +--> More: https://etherpad.openstack.org/p/tripleo-upgrade-squad-status
>
> +---+
> | Containers |
> +---+
>
> +--> Good progress on upgrades, now working on THT tasks to upgrade
> undercloud services
> +--> Focusing on UX problems: logs, permissions, directories, complete
> deployment message
> +--> Container workflow is still work in progress, and needed to make
> progress on CI / container updates
> +--> We had to revert containerized undercloud testing on fs010 :
> https://bugs.launchpad.net/tripleo/+bug/1762422
> +--> More:
> https://etherpad.openstack.org/p/tripleo-containers-squad-status
>
> +--+
> | config-download |
> +--+
>
> +--> Moving to config-download by default is imminent.
> +--> ceph/octavia/skydive migration is wip.
> +--> Inventory improvements in progress.
> +--> Polishing tripleo-common deploy_plan and messaging patches to get
> correct deployment state tracking.
> +--> UI work is work in progress.
> +--> More:
> https://etherpad.openstack.org/p/tripleo-config-download-squad-status
>
> +--+
> | Integration |
> +--+
>
> +--> Migrate to new ceph-ansible container images naming style.
> +--> Config-download transition is still ongoing.
> +--> More:
> https://etherpad.openstack.org/p/tripleo-integration-squad-status
>
> +-+
> | UI/CLI |
> +-+
>
> +--> Efforts on config-download integration
> +--> Investigating undeploy_plan workflow in tripleo-common
> +--> Maintaining pending UI patches to be up to date with tripleo-common
> changes
> +--> More: https://etherpad.openstack.org/p/tripleo-ui-cli-squad-status
>
> +---+
> | Validations |
> +---+
>
> +--> OpenShift on OpenStack validations in progress
> +--> Starting work on Custom validations/swift storage
> +--> Need reviews, see etherpad
> +--> More:
> https://etherpad.openstack.org/p/tripleo-validations-squad-status
>
> +---+
> | Networking |
> +---+
>
> +--> No updates this week.
> +--> More:
> https://etherpad.openstack.org/p/tripleo-networking-squad-status
>
> +--+
> | Workflows |
> +--+
>
> +--> Need reviews, see etherpad.
> +--> More: https://etherpad.openstack.org/p/tripleo-workflows-squad-status
>
> +---+
> | Security |
> +---+
>
> +--> Last meeting's was about Public TLS by default, Limit TripleO users
> and Security Hardening.
> +--> More: https://etherpad.openstack.org/p/tripleo-security-squad
>
> ++
> | Owl fact  |
> ++
>
> Did you know owls were good hunters? Check this video:
> https://youtu.be/a68fIQzaDBY?t=39
> Don't mess with owls ;-)
>
> Thanks all for reading and stay tuned!
> --
> Your fellow reporter, Emilien Macchi
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [tripleo][ci] use of tags in launchpad bugs

[Wesley Hayutin]

FYI...

This is news to me so thanks to Emilien for pointing it out [1].
There are official tags for tripleo launchpad bugs.  Personally, I like
what I've seen recently with some extra tags as they could be helpful in
finding the history of particular issues.
So hypothetically would it be "wrong" to create an official tag for each
featureset config number upstream.  I ask because that is adding a lot of
tags but also serves as a good test case for what is good/bad use of tags.

Thanks

[1] https://bugs.launchpad.net/tripleo/+manage-official-tags
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Fwd: [tripleo] PTG session about All-In-One installer: recap & roadmap

[Wesley Hayutin]

On Thu, 5 Apr 2018 at 12:25 Emilien Macchi  wrote:

> On Thu, Apr 5, 2018 at 4:37 AM, Dan Prince  wrote:
>
> Much of the work on this is already there. We've been using this stuff
>> for over a year to dev/test the containerization efforts for a long
>> time now (and thanks for your help with this effort). The problem I
>> think is how it is all packaged. While you can use it today it
>> involves some tricks (docker in docker), or requires you to use an
>> extra VM to minimize the installation footprint on your laptop.
>>
>> Much of the remaining work here is really just about packaging and
>> technical debt. If we put tripleoclient and heat-monolith into a
>> container that solves much of the requirements problems and
>> essentially gives you a container which can transform Heat templates
>> to Ansible. From the ansible side we need to do a bit more work to
>> mimimize our dependencies (i.e. heat hooks). Using a virtual-env would
>> be one option for developers if we could make that work. I lighter set
>> of RPM packages would be another way to do it. Perhaps both...
>> Then a smaller wrapper around these things (which I personally would
>> like to name) to make it all really tight.
>
>
> So if I summarize the discussion:
>
> - A lot of positive feedback about the idea and many use cases, which is
> great.
>
> - Support for non-containerized services is not required, as long as we
> provide a way to update containers with under-review patches for developers.
>

Hrm..  I was just speaking to Alfredo about this.  We may need to have a
better understanding of the various ecosystems where TripleO is in play
here to have a fully informed decision.
By ecosystem I'm referring to RDO, centos, and upstream and the containers
used in deployments.  I suspect a non-containerized deployment may still be
required, but looking for the packaging team to weigh in.


>
> - We'll probably want to breakdown the "openstack undercloud deploy"
> process into pieces
> * start an ephemeral Heat container
> * create the Heat stack passing all requested -e's
> * run config-download and save the output
>
> And then remove undercloud specific logic, so we can provide a generic
> way to create the config-download playbooks.
> This generic way would be consumed by the undercloud deploy commands but
> also by the new all-in-one wrapper.
>
> - Speaking of the wrapper, we will probably have a new one. Several names
> were proposed:
> * openstack tripleo deploy
> * openstack talon deploy
> * openstack elf deploy
>
> - The wrapper would work with deployed-server, so we would noop Neutron
> networks and use fixed IPs.
>
> - Investigate the packaging work: containerize tripleoclient and
> dependencies, see how we can containerized Ansible + dependencies (and
> eventually reduce them at strict minimum).
>
> Let me know if I missed something important, hopefully we can move things
> forward during this cycle.
> --
> Emilien Macchi
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo] PTG session about All-In-One installer: recap & roadmap

[Wesley Hayutin]

On Tue, 3 Apr 2018 at 13:53 Dan Prince  wrote:

> On Tue, Apr 3, 2018 at 10:00 AM, Javier Pena  wrote:
> >
> >> Greeting folks,
> >>
> >> During the last PTG we spent time discussing some ideas around an
> All-In-One
> >> installer, using 100% of the TripleO bits to deploy a single node
> OpenStack
> >> very similar with what we have today with the containerized undercloud
> and
> >> what we also have with other tools like Packstack or Devstack.
> >>
> >> https://etherpad.openstack.org/p/tripleo-rocky-all-in-one
> >>
> >
> > I'm really +1 to this. And as a Packstack developer, I'd love to see
> this as a
> > mid-term Packstack replacement. So let's dive into the details.
>
> Curious on this one actually, do you see a need for continued
> baremetal support? Today we support both baremetal and containers.
> Perhaps "support" is a strong word. We support both in terms of
> installation but only containers now have fully supported upgrades.
>
> The interfaces we have today still support baremetal and containers
> but there were some suggestions about getting rid of baremetal support
> and only having containers. If we were to remove baremetal support
> though, Could we keep the Packstack case intact by just using
> containers instead?
>
> Dan
>

Hey couple thoughts..
1.  I've added this topic to the RDO meeting tomorrow.
2.  Just a thought, the "elf owl" is the worlds smallest owl at least
according to the internets   Maybe the all in one could be nick named
tripleo elf?  Talon is cool too.
3.  From a CI perspective, I see this being very help with:
  a: faster run times generally, but especially for an upgrade tests.  It
may be possible to have upgrades gating tripleo projects again.
  b: enabling more packaging tests to be done with TripleO
  c: If developers dig it, we have a better chance at getting TripleO into
other project's check jobs / third party jobs where current requirements
and run times are prohibitive.
  d: Generally speaking replacing packstack / devstack in devel and CI
workflows  where it still exists.
  e: Improved utilization of our resources in RDO-Cloud

It would be interesting to me to see more design and a little more thought
into the potential use cases before we get far along.  Looks like there is
a good start to that here [2].
I'll add some comments with the potential use cases for CI.

/me is very happy to see this moving! Thanks all

[1] https://en.wikipedia.org/wiki/Elf_owl
[2]
https://review.openstack.org/#/c/547038/1/doc/source/install/advanced_deployment/all_in_one.rst



>
> >
> >> One of the problems that we're trying to solve here is to give a simple
> tool
> >> for developers so they can both easily and quickly deploy an OpenStack
> for
> >> their needs.
> >>
> >> "As a developer, I need to deploy OpenStack in a VM on my laptop,
> quickly and
> >> without complexity, reproducing the same exact same tooling as TripleO
> is
> >> using."
> >> "As a Neutron developer, I need to develop a feature in Neutron and
> test it
> >> with TripleO in my local env."
> >> "As a TripleO dev, I need to implement a new service and test its
> deployment
> >> in my local env."
> >> "As a developer, I need to reproduce a bug in TripleO CI that blocks the
> >> production chain, quickly and simply."
> >>
> >
> > "As a packager, I want an easy/low overhead way to test updated packages
> with TripleO bits, so I can make sure they will not break any automation".
> >
> >> Probably more use cases, but to me that's what came into my mind now.
> >>
> >> Dan kicked-off a doc patch a month ago:
> >> https://review.openstack.org/#/c/547038/
> >> And I just went ahead and proposed a blueprint:
> >> https://blueprints.launchpad.net/tripleo/+spec/all-in-one
> >> So hopefully we can start prototyping something during Rocky.
> >>
> >> Before talking about the actual implementation, I would like to gather
> >> feedback from people interested by the use-cases. If you recognize
> yourself
> >> in these use-cases and you're not using TripleO today to test your
> things
> >> because it's too complex to deploy, we want to hear from you.
> >> I want to see feedback (positive or negative) about this idea. We need
> to
> >> gather ideas, use cases, needs, before we go design a prototype in
> Rocky.
> >>
> >
> > I would like to offer help with initial testing once there is something
> in the repos, so count me in!
> >
> > Regards,
> > Javier
> >
> >> Thanks everyone who'll be involved,
> >> --
> >> Emilien Macchi
> >>
> >>
> __
> >> OpenStack Development Mailing List (not for usage questions)
> >> Unsubscribe:
> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
> >
> __
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe:
> openstack-dev-requ...@lists.openstack.org?

Re: [openstack-dev] [tripleo] PTG session about All-In-One installer: recap & roadmap

[Wesley Hayutin]

On Tue, 3 Apr 2018 at 10:00 Javier Pena  wrote:

>
> > Greeting folks,
> >
> > During the last PTG we spent time discussing some ideas around an
> All-In-One
> > installer, using 100% of the TripleO bits to deploy a single node
> OpenStack
> > very similar with what we have today with the containerized undercloud
> and
> > what we also have with other tools like Packstack or Devstack.
> >
> > https://etherpad.openstack.org/p/tripleo-rocky-all-in-one
> >
>
> I'm really +1 to this. And as a Packstack developer, I'd love to see this
> as a
> mid-term Packstack replacement. So let's dive into the details.
>
> > One of the problems that we're trying to solve here is to give a simple
> tool
> > for developers so they can both easily and quickly deploy an OpenStack
> for
> > their needs.
> >
> > "As a developer, I need to deploy OpenStack in a VM on my laptop,
> quickly and
> > without complexity, reproducing the same exact same tooling as TripleO is
> > using."
> > "As a Neutron developer, I need to develop a feature in Neutron and test
> it
> > with TripleO in my local env."
> > "As a TripleO dev, I need to implement a new service and test its
> deployment
> > in my local env."
> > "As a developer, I need to reproduce a bug in TripleO CI that blocks the
> > production chain, quickly and simply."
> >
>
> "As a packager, I want an easy/low overhead way to test updated packages
> with TripleO bits, so I can make sure they will not break any automation".
>

I suspect we need to not only update packages, but also update containers,
wdyt?


>
> > Probably more use cases, but to me that's what came into my mind now.
> >
> > Dan kicked-off a doc patch a month ago:
> > https://review.openstack.org/#/c/547038/
> > And I just went ahead and proposed a blueprint:
> > https://blueprints.launchpad.net/tripleo/+spec/all-in-one
> > So hopefully we can start prototyping something during Rocky.
> >
> > Before talking about the actual implementation, I would like to gather
> > feedback from people interested by the use-cases. If you recognize
> yourself
> > in these use-cases and you're not using TripleO today to test your things
> > because it's too complex to deploy, we want to hear from you.
> > I want to see feedback (positive or negative) about this idea. We need to
> > gather ideas, use cases, needs, before we go design a prototype in Rocky.
> >
>
> I would like to offer help with initial testing once there is something in
> the repos, so count me in!
>
> Regards,
> Javier
>
> > Thanks everyone who'll be involved,
> > --
> > Emilien Macchi
> >
> >
> __
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe:
> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo] PTG session about All-In-One installer: recap & roadmap

[Wesley Hayutin]

On Fri, 30 Mar 2018 at 08:45 Paul Grist  wrote:

> On Thu, Mar 29, 2018 at 5:32 PM, Emilien Macchi 
> wrote:
>
>> Greeting folks,
>>
>> During the last PTG we spent time discussing some ideas around an
>> All-In-One installer, using 100% of the TripleO bits to deploy a single
>> node OpenStack very similar with what we have today with the containerized
>> undercloud and what we also have with other tools like Packstack or
>> Devstack.
>>
>> https://etherpad.openstack.org/p/tripleo-rocky-all-in-one
>>
>> One of the problems that we're trying to solve here is to give a simple
>> tool for developers so they can both easily and quickly deploy an OpenStack
>> for their needs.
>>
>> Big +1 on the concept,  thanks to all those putting effort into this.
>
>
>> "As a developer, I need to deploy OpenStack in a VM on my laptop, quickly
>> and without complexity, reproducing the same exact same tooling as TripleO
>> is using."
>> "As a Neutron developer, I need to develop a feature in Neutron and test
>> it with TripleO in my local env."
>> "As a TripleO dev, I need to implement a new service and test its
>> deployment in my local env."
>> "As a developer, I need to reproduce a bug in TripleO CI that blocks the
>> production chain, quickly and simply."
>>
>>
> Would this also be an opportunity for CI to enable lighter weight sanity
> and preliminary tests?
> "As a project, I want to implement a TripleO CI gate to detect regressions
> early, but have resource and test execution time constraints"
>
>
Paul,
You are 100% correct sir.  That is the opportunity and intention we have
here.  Moving forward I see a single node installer that is comprable to
devstack/packstack as a requirement for the project as we continue to
improve the deployment to enable other projects to test/ci with TripleO in
their check jobs.

Thanks for responding and your support!


>
>
>> Probably more use cases, but to me that's what came into my mind now.
>>
>> Dan kicked-off a doc patch a month ago:
>> https://review.openstack.org/#/c/547038/
>> And I just went ahead and proposed a blueprint:
>> https://blueprints.launchpad.net/tripleo/+spec/all-in-one
>> So hopefully we can start prototyping something during Rocky.
>>
>> Before talking about the actual implementation, I would like to gather
>> feedback from people interested by the use-cases. If you recognize yourself
>> in these use-cases and you're not using TripleO today to test your things
>> because it's too complex to deploy, we want to hear from you.
>> I want to see feedback (positive or negative) about this idea. We need to
>> gather ideas, use cases, needs, before we go design a prototype in Rocky.
>>
>> Thanks everyone who'll be involved,
>> --
>> Emilien Macchi
>>
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev