Re: [openstack-dev] Zuul v3 - What's Coming: What to expect with the Zuul v3 Rollout

[James E. Blair]

"bogda...@mail.ru"  writes:

> That's great news! In-repo configs will speed up development for teams,
> with a security caveat for infrastructure team to keep in mind. The
> ansible runner CI node which runs playbooks for defined jobs, should not
> content sensitive information, like keys and secrets in files or
> exported env vars, unless they are a one time or limited in time. The
> same applies to the nodepool nodes allocated for a particular CI test
> run. Otherwise, a malformed patch could make ansible to cat/echo all of
> the secrets to the publicly available build logs.

Indeed that is a risk.  To mitigate that, we are building a restricted
execution environment for Ansible so that jobs defined in-repo will only
be allowed to access a per-job staging area on the runner.  And we also
plan on running that in a chrooted container.

These protections are not complete yet, which is why our test instance
at the moment is very limited in scope.

-Jim

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Zuul v3 - What's Coming: What to expect with the Zuul v3 Rollout

[bogda...@mail.ru]

That's great news! In-repo configs will speed up development for teams,
with a security caveat for infrastructure team to keep in mind. The
ansible runner CI node which runs playbooks for defined jobs, should not
content sensitive information, like keys and secrets in files or
exported env vars, unless they are a one time or limited in time. The
same applies to the nodepool nodes allocated for a particular CI test
run. Otherwise, a malformed patch could make ansible to cat/echo all of
the secrets to the publicly available build logs.

> 
> From: Monty Taylor [mordred at inaugust.com]
> Sent: 01 March 2017 7:26
> To: OpenStack Development Mailing List (not for usage questions)
> Subject: [openstack-dev] Zuul v3 - What's Coming: What to expect with
the   Zuul v3 Rollout
>
> ...
> * Self-testing In-Repo Job Config
> * Ansible Job Content
> ...

-- 
Best regards,
Bogdan Dobrelya,
Irc #bogdando

-- 
Best regards,
Bogdan Dobrelya,
Irc #bogdando

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Zuul v3 - What's Coming: What to expect with the Zuul v3 Rollout

[Joshua Hesketh]

Thanks for the great write-up Monty :-). Last week was great fun and zuulv3
is making excellent process. I'm excited for the switch-over.

Cheers,
Josh

On Wed, Mar 1, 2017 at 10:26 AM, Monty Taylor  wrote:

> Hi everybody!
>
> This content can also be found at
> http://inaugust.com/posts/whats-coming-zuulv3.html - but I've pasted it
> in here directly because I know that some folks don't like clicking links.
>
> tl;dr - At last week's OpenStack PTG, the OpenStack Infra team ran the
> first Zuul v3 job, so it's time to start getting everybody ready for
> what's coming
>
> **Don't Panic!** Awesome changes are coming, but you are NOT on the hook
> for rewriting all of your project's gate jobs or anything crazy like
> that. Now grab a seat by the fire, pour yourself a drink while I spin a
> yarn about days gone by and days yet to come.
>
> First, some background
>
> The OpenStack Infra team has been hard at work for quite a while on a
> new version of Zuul (where by 'quite some time' I mean that Jim Blair
> and I had our first Zuul v3 design whiteboarding session in 2014). As
> you might be able to guess given the amount of time, there are some big
> things coming that will have a real and visible impact on the OpenStack
> community and beyond. Since we have a running Zuul v3 now [1], it seemed
> like the time to start getting folks up to speed on what to expect.
>
> There is other deep-dive information on architecture and rationale if
> you're interested[2], but for now we'll focus on what's relevant for end
> users. We're also going to start sending out a bi-weekly "Status of Zuul
> v3" email to the openstack-dev@lists.openstack.org mailing list ... so
> stay tuned!
>
> **Important Note** This post includes some code snippets - but v3 is
> still a work in progress. We know of at least one breaking change that
> is coming to the config format, so please treat this not as a tutorial,
> but as a conceptual overview. Syntax is subject to change.
>
> The Big Ticket Items
>
> While there are a bunch of changes behind the scenes, there are a
> reasonably tractable number of user-facing differences.
>
> * Self-testing In-Repo Job Config
> * Ansible Job Content
> * First-class Multi-node Jobs
> * Improved Job Reuse
> * Support for non-OpenStack Code and Node Systems
> * and Much, Much More
>
> Self-testing In-Repo Job Config
>
> This is probably the biggest deal. There are a lot of OpenStack Devs
> (around 2k in Ocata) and a lot of repositories (1689) There a lot fewer
> folks on the project-config-core team who are the ones who review all of
> the job config changes (please everyone thank Andreas Jaeger next time
> you see him). That's not awesome.
>
> Self-testing in-repo job config is awesome.
>
> Many systems out there these days have an in-repo job config system.
> Travis CI has had it since day one, and Jenkins has recently added
> support for a Jenkinsfile inside of git repos. With Zuul v3, we'll have
> it too.
>
> Once we roll out v3 to everyone, as a supplement to jobs defined in our
> central config repositories, each project will be able to add a
> zuul.yaml file to their own repo:
>
>
> - job:
> name: my_awesome_job
> nodes:
>   - name: controller
> label: centos-7
>
> - project:
> name: openstack/awesome_project
> check:
>   jobs:
> - my_awesome_job
>
> It's a small file, but there is a lot going on, so let's unpack it.
>
> First we define a job to run. It's named my_awesome_job and it needs one
> node. That node will be named controller and will be based on the
> centos-7 base node in nodepool.
>
> In the next section, we say that we want to run that job in the check
> pipeline, which in OpenStack is defined as the jobs that run when
> patchsets are proposed.
>
> And it's also self-testing!
>
> Everyone knows the fun game of writing a patch to the test jobs, getting
> it approved, then hoping it works once it starts running. With Zuul v3
> in-repo jobs, if there is a change to job definitions in a proposed
> patch, that patch will be tested with those changes applied. And since
> it's Zuul, Depends-On footers are honored as well - so iteration on
> getting a test job right becomes just like iterating on any other patch
> or sequence of patches.
>
> Ansible Job Content
>
> The job my_awesome_job isn't very useful if it doesn't define any
> content. That's done in the repo as well, in playbooks/my_awesome_job.yaml:
>
>
> - hosts: controller
>   tasks:
> - name: Run make tests
>   shell: make distcheck
>
> As previously mentioned, the job content is now defined in Ansible
> rather than using our Jenkins Job Builder tool. This playbook is going
> to run a tasks on a host called controller which you may remember we
> requested in the job definition. On that host, it will run make
> distcheck. Pretty much anything you can do in Ansible, you can do in a
> Zuul job now, and the playbooks should also be re-usable outside of a
> testing 

Re: [openstack-dev] Zuul v3 - What's Coming: What to expect with the Zuul v3 Rollout

[Emilien Macchi]

On Wed, Mar 1, 2017 at 12:31 PM, Kosnik, Lubosz <lubosz.kos...@intel.com> wrote:
> So did I understand that properly. There will be possibility to create real
> multi-node tests like with 3-4 nodes?

You can already do that, openstack-infra/nodepool proposes some
2-nodes, 3-nodes and even 4-nodes jobs. See jenkins/jobs/projects.yaml
to see which projects are using it.

> Cheers,
> Lubosz
>
> On Feb 28, 2017, at 7:13 PM, joehuang <joehu...@huawei.com> wrote:
>
> So cool! Look forward to multi-node jobs as first class
>
> Best Regards
> Chaoyi Huang (joehuang)
>
> 
> From: Monty Taylor [mord...@inaugust.com]
> Sent: 01 March 2017 7:26
> To: OpenStack Development Mailing List (not for usage questions)
> Subject: [openstack-dev] Zuul v3 - What's Coming: What to expect with the
> Zuul v3 Rollout
>
> Hi everybody!
>
> This content can also be found at
> http://inaugust.com/posts/whats-coming-zuulv3.html - but I've pasted it
> in here directly because I know that some folks don't like clicking links.
>
> tl;dr - At last week's OpenStack PTG, the OpenStack Infra team ran the
> first Zuul v3 job, so it's time to start getting everybody ready for
> what's coming
>
> **Don't Panic!** Awesome changes are coming, but you are NOT on the hook
> for rewriting all of your project's gate jobs or anything crazy like
> that. Now grab a seat by the fire, pour yourself a drink while I spin a
> yarn about days gone by and days yet to come.
>
> First, some background
>
> The OpenStack Infra team has been hard at work for quite a while on a
> new version of Zuul (where by 'quite some time' I mean that Jim Blair
> and I had our first Zuul v3 design whiteboarding session in 2014). As
> you might be able to guess given the amount of time, there are some big
> things coming that will have a real and visible impact on the OpenStack
> community and beyond. Since we have a running Zuul v3 now [1], it seemed
> like the time to start getting folks up to speed on what to expect.
>
> There is other deep-dive information on architecture and rationale if
> you're interested[2], but for now we'll focus on what's relevant for end
> users. We're also going to start sending out a bi-weekly "Status of Zuul
> v3" email to the openstack-dev@lists.openstack.org mailing list ... so
> stay tuned!
>
> **Important Note** This post includes some code snippets - but v3 is
> still a work in progress. We know of at least one breaking change that
> is coming to the config format, so please treat this not as a tutorial,
> but as a conceptual overview. Syntax is subject to change.
>
> The Big Ticket Items
>
> While there are a bunch of changes behind the scenes, there are a
> reasonably tractable number of user-facing differences.
>
> * Self-testing In-Repo Job Config
> * Ansible Job Content
> * First-class Multi-node Jobs
> * Improved Job Reuse
> * Support for non-OpenStack Code and Node Systems
> * and Much, Much More
>
> Self-testing In-Repo Job Config
>
> This is probably the biggest deal. There are a lot of OpenStack Devs
> (around 2k in Ocata) and a lot of repositories (1689) There a lot fewer
> folks on the project-config-core team who are the ones who review all of
> the job config changes (please everyone thank Andreas Jaeger next time
> you see him). That's not awesome.
>
> Self-testing in-repo job config is awesome.
>
> Many systems out there these days have an in-repo job config system.
> Travis CI has had it since day one, and Jenkins has recently added
> support for a Jenkinsfile inside of git repos. With Zuul v3, we'll have
> it too.
>
> Once we roll out v3 to everyone, as a supplement to jobs defined in our
> central config repositories, each project will be able to add a
> zuul.yaml file to their own repo:
>
>
> - job:
>name: my_awesome_job
>nodes:
>  - name: controller
>label: centos-7
>
> - project:
>name: openstack/awesome_project
>check:
>  jobs:
>- my_awesome_job
>
> It's a small file, but there is a lot going on, so let's unpack it.
>
> First we define a job to run. It's named my_awesome_job and it needs one
> node. That node will be named controller and will be based on the
> centos-7 base node in nodepool.
>
> In the next section, we say that we want to run that job in the check
> pipeline, which in OpenStack is defined as the jobs that run when
> patchsets are proposed.
>
> And it's also self-testing!
>
> Everyone knows the fun game of writing a patch to the test jobs, getting
> it approved, then hoping it works once it starts running. With Zuul v3
> in-repo jobs, if there is a change to job definiti

Re: [openstack-dev] Zuul v3 - What's Coming: What to expect with the Zuul v3 Rollout

[Clark Boylan]

Yes. It is also worth noting you can do this today too and some
projects/tests do.

Clark

On Wed, Mar 1, 2017, at 09:31 AM, Kosnik, Lubosz wrote:
> So did I understand that properly. There will be possibility to create
> real multi-node tests like with 3-4 nodes?
> 
> Cheers,
> Lubosz
> 
> On Feb 28, 2017, at 7:13 PM, joehuang
> <joehu...@huawei.com<mailto:joehu...@huawei.com>> wrote:
> 
> So cool! Look forward to multi-node jobs as first class
> 
> Best Regards
> Chaoyi Huang (joehuang)
> 
> 
> From: Monty Taylor [mord...@inaugust.com<mailto:mord...@inaugust.com>]
> Sent: 01 March 2017 7:26
> To: OpenStack Development Mailing List (not for usage questions)
> Subject: [openstack-dev] Zuul v3 - What's Coming: What to expect with the
>   Zuul v3 Rollout
> 
> Hi everybody!
> 
> This content can also be found at
> http://inaugust.com/posts/whats-coming-zuulv3.html - but I've pasted it
> in here directly because I know that some folks don't like clicking
> links.
> 
> tl;dr - At last week's OpenStack PTG, the OpenStack Infra team ran the
> first Zuul v3 job, so it's time to start getting everybody ready for
> what's coming
> 
> **Don't Panic!** Awesome changes are coming, but you are NOT on the hook
> for rewriting all of your project's gate jobs or anything crazy like
> that. Now grab a seat by the fire, pour yourself a drink while I spin a
> yarn about days gone by and days yet to come.
> 
> First, some background
> 
> The OpenStack Infra team has been hard at work for quite a while on a
> new version of Zuul (where by 'quite some time' I mean that Jim Blair
> and I had our first Zuul v3 design whiteboarding session in 2014). As
> you might be able to guess given the amount of time, there are some big
> things coming that will have a real and visible impact on the OpenStack
> community and beyond. Since we have a running Zuul v3 now [1], it seemed
> like the time to start getting folks up to speed on what to expect.
> 
> There is other deep-dive information on architecture and rationale if
> you're interested[2], but for now we'll focus on what's relevant for end
> users. We're also going to start sending out a bi-weekly "Status of Zuul
> v3" email to the
> openstack-dev@lists.openstack.org<mailto:openstack-dev@lists.openstack.org>
> mailing list ... so
> stay tuned!
> 
> **Important Note** This post includes some code snippets - but v3 is
> still a work in progress. We know of at least one breaking change that
> is coming to the config format, so please treat this not as a tutorial,
> but as a conceptual overview. Syntax is subject to change.
> 
> The Big Ticket Items
> 
> While there are a bunch of changes behind the scenes, there are a
> reasonably tractable number of user-facing differences.
> 
> * Self-testing In-Repo Job Config
> * Ansible Job Content
> * First-class Multi-node Jobs
> * Improved Job Reuse
> * Support for non-OpenStack Code and Node Systems
> * and Much, Much More
> 
> Self-testing In-Repo Job Config
> 
> This is probably the biggest deal. There are a lot of OpenStack Devs
> (around 2k in Ocata) and a lot of repositories (1689) There a lot fewer
> folks on the project-config-core team who are the ones who review all of
> the job config changes (please everyone thank Andreas Jaeger next time
> you see him). That's not awesome.
> 
> Self-testing in-repo job config is awesome.
> 
> Many systems out there these days have an in-repo job config system.
> Travis CI has had it since day one, and Jenkins has recently added
> support for a Jenkinsfile inside of git repos. With Zuul v3, we'll have
> it too.
> 
> Once we roll out v3 to everyone, as a supplement to jobs defined in our
> central config repositories, each project will be able to add a
> zuul.yaml file to their own repo:
> 
> 
> - job:
>name: my_awesome_job
>nodes:
>  - name: controller
>label: centos-7
> 
> - project:
>name: openstack/awesome_project
>check:
>  jobs:
>- my_awesome_job
> 
> It's a small file, but there is a lot going on, so let's unpack it.
> 
> First we define a job to run. It's named my_awesome_job and it needs one
> node. That node will be named controller and will be based on the
> centos-7 base node in nodepool.
> 
> In the next section, we say that we want to run that job in the check
> pipeline, which in OpenStack is defined as the jobs that run when
> patchsets are proposed.
> 
> And it's also self-testing!
> 
> Everyone knows the fun game of writing a patch to the test jobs, getting
> it approved, then hoping it works once it starts running. W

Re: [openstack-dev] Zuul v3 - What's Coming: What to expect with the Zuul v3 Rollout

[Kosnik, Lubosz]

So did I understand that properly. There will be possibility to create real 
multi-node tests like with 3-4 nodes?

Cheers,
Lubosz

On Feb 28, 2017, at 7:13 PM, joehuang 
<joehu...@huawei.com<mailto:joehu...@huawei.com>> wrote:

So cool! Look forward to multi-node jobs as first class

Best Regards
Chaoyi Huang (joehuang)


From: Monty Taylor [mord...@inaugust.com<mailto:mord...@inaugust.com>]
Sent: 01 March 2017 7:26
To: OpenStack Development Mailing List (not for usage questions)
Subject: [openstack-dev] Zuul v3 - What's Coming: What to expect with the   
Zuul v3 Rollout

Hi everybody!

This content can also be found at
http://inaugust.com/posts/whats-coming-zuulv3.html - but I've pasted it
in here directly because I know that some folks don't like clicking links.

tl;dr - At last week's OpenStack PTG, the OpenStack Infra team ran the
first Zuul v3 job, so it's time to start getting everybody ready for
what's coming

**Don't Panic!** Awesome changes are coming, but you are NOT on the hook
for rewriting all of your project's gate jobs or anything crazy like
that. Now grab a seat by the fire, pour yourself a drink while I spin a
yarn about days gone by and days yet to come.

First, some background

The OpenStack Infra team has been hard at work for quite a while on a
new version of Zuul (where by 'quite some time' I mean that Jim Blair
and I had our first Zuul v3 design whiteboarding session in 2014). As
you might be able to guess given the amount of time, there are some big
things coming that will have a real and visible impact on the OpenStack
community and beyond. Since we have a running Zuul v3 now [1], it seemed
like the time to start getting folks up to speed on what to expect.

There is other deep-dive information on architecture and rationale if
you're interested[2], but for now we'll focus on what's relevant for end
users. We're also going to start sending out a bi-weekly "Status of Zuul
v3" email to the 
openstack-dev@lists.openstack.org<mailto:openstack-dev@lists.openstack.org> 
mailing list ... so
stay tuned!

**Important Note** This post includes some code snippets - but v3 is
still a work in progress. We know of at least one breaking change that
is coming to the config format, so please treat this not as a tutorial,
but as a conceptual overview. Syntax is subject to change.

The Big Ticket Items

While there are a bunch of changes behind the scenes, there are a
reasonably tractable number of user-facing differences.

* Self-testing In-Repo Job Config
* Ansible Job Content
* First-class Multi-node Jobs
* Improved Job Reuse
* Support for non-OpenStack Code and Node Systems
* and Much, Much More

Self-testing In-Repo Job Config

This is probably the biggest deal. There are a lot of OpenStack Devs
(around 2k in Ocata) and a lot of repositories (1689) There a lot fewer
folks on the project-config-core team who are the ones who review all of
the job config changes (please everyone thank Andreas Jaeger next time
you see him). That's not awesome.

Self-testing in-repo job config is awesome.

Many systems out there these days have an in-repo job config system.
Travis CI has had it since day one, and Jenkins has recently added
support for a Jenkinsfile inside of git repos. With Zuul v3, we'll have
it too.

Once we roll out v3 to everyone, as a supplement to jobs defined in our
central config repositories, each project will be able to add a
zuul.yaml file to their own repo:


- job:
   name: my_awesome_job
   nodes:
 - name: controller
   label: centos-7

- project:
   name: openstack/awesome_project
   check:
 jobs:
   - my_awesome_job

It's a small file, but there is a lot going on, so let's unpack it.

First we define a job to run. It's named my_awesome_job and it needs one
node. That node will be named controller and will be based on the
centos-7 base node in nodepool.

In the next section, we say that we want to run that job in the check
pipeline, which in OpenStack is defined as the jobs that run when
patchsets are proposed.

And it's also self-testing!

Everyone knows the fun game of writing a patch to the test jobs, getting
it approved, then hoping it works once it starts running. With Zuul v3
in-repo jobs, if there is a change to job definitions in a proposed
patch, that patch will be tested with those changes applied. And since
it's Zuul, Depends-On footers are honored as well - so iteration on
getting a test job right becomes just like iterating on any other patch
or sequence of patches.

Ansible Job Content

The job my_awesome_job isn't very useful if it doesn't define any
content. That's done in the repo as well, in playbooks/my_awesome_job.yaml:


- hosts: controller
 tasks:
   - name: Run make tests
 shell: make distcheck

As previously mentioned, the job content is now defined in Ansible
rather than using our Jenkins Job Builder tool. This playbook is going
to run a tasks on a host called co

Re: [openstack-dev] Zuul v3 - What's Coming: What to expect with the Zuul v3 Rollout

[joehuang]

So cool! Look forward to multi-node jobs as first class

Best Regards
Chaoyi Huang (joehuang)


From: Monty Taylor [mord...@inaugust.com]
Sent: 01 March 2017 7:26
To: OpenStack Development Mailing List (not for usage questions)
Subject: [openstack-dev] Zuul v3 - What's Coming: What to expect with the   
Zuul v3 Rollout

Hi everybody!

This content can also be found at
http://inaugust.com/posts/whats-coming-zuulv3.html - but I've pasted it
in here directly because I know that some folks don't like clicking links.

tl;dr - At last week's OpenStack PTG, the OpenStack Infra team ran the
first Zuul v3 job, so it's time to start getting everybody ready for
what's coming

**Don't Panic!** Awesome changes are coming, but you are NOT on the hook
for rewriting all of your project's gate jobs or anything crazy like
that. Now grab a seat by the fire, pour yourself a drink while I spin a
yarn about days gone by and days yet to come.

First, some background

The OpenStack Infra team has been hard at work for quite a while on a
new version of Zuul (where by 'quite some time' I mean that Jim Blair
and I had our first Zuul v3 design whiteboarding session in 2014). As
you might be able to guess given the amount of time, there are some big
things coming that will have a real and visible impact on the OpenStack
community and beyond. Since we have a running Zuul v3 now [1], it seemed
like the time to start getting folks up to speed on what to expect.

There is other deep-dive information on architecture and rationale if
you're interested[2], but for now we'll focus on what's relevant for end
users. We're also going to start sending out a bi-weekly "Status of Zuul
v3" email to the openstack-dev@lists.openstack.org mailing list ... so
stay tuned!

**Important Note** This post includes some code snippets - but v3 is
still a work in progress. We know of at least one breaking change that
is coming to the config format, so please treat this not as a tutorial,
but as a conceptual overview. Syntax is subject to change.

The Big Ticket Items

While there are a bunch of changes behind the scenes, there are a
reasonably tractable number of user-facing differences.

* Self-testing In-Repo Job Config
* Ansible Job Content
* First-class Multi-node Jobs
* Improved Job Reuse
* Support for non-OpenStack Code and Node Systems
* and Much, Much More

Self-testing In-Repo Job Config

This is probably the biggest deal. There are a lot of OpenStack Devs
(around 2k in Ocata) and a lot of repositories (1689) There a lot fewer
folks on the project-config-core team who are the ones who review all of
the job config changes (please everyone thank Andreas Jaeger next time
you see him). That's not awesome.

Self-testing in-repo job config is awesome.

Many systems out there these days have an in-repo job config system.
Travis CI has had it since day one, and Jenkins has recently added
support for a Jenkinsfile inside of git repos. With Zuul v3, we'll have
it too.

Once we roll out v3 to everyone, as a supplement to jobs defined in our
central config repositories, each project will be able to add a
zuul.yaml file to their own repo:


- job:
name: my_awesome_job
nodes:
  - name: controller
label: centos-7

- project:
name: openstack/awesome_project
check:
  jobs:
- my_awesome_job

It's a small file, but there is a lot going on, so let's unpack it.

First we define a job to run. It's named my_awesome_job and it needs one
node. That node will be named controller and will be based on the
centos-7 base node in nodepool.

In the next section, we say that we want to run that job in the check
pipeline, which in OpenStack is defined as the jobs that run when
patchsets are proposed.

And it's also self-testing!

Everyone knows the fun game of writing a patch to the test jobs, getting
it approved, then hoping it works once it starts running. With Zuul v3
in-repo jobs, if there is a change to job definitions in a proposed
patch, that patch will be tested with those changes applied. And since
it's Zuul, Depends-On footers are honored as well - so iteration on
getting a test job right becomes just like iterating on any other patch
or sequence of patches.

Ansible Job Content

The job my_awesome_job isn't very useful if it doesn't define any
content. That's done in the repo as well, in playbooks/my_awesome_job.yaml:


- hosts: controller
  tasks:
- name: Run make tests
  shell: make distcheck

As previously mentioned, the job content is now defined in Ansible
rather than using our Jenkins Job Builder tool. This playbook is going
to run a tasks on a host called controller which you may remember we
requested in the job definition. On that host, it will run make
distcheck. Pretty much anything you can do in Ansible, you can do in a
Zuul job now, and the playbooks should also be re-usable outside of a
testing context.

First Class Multi-Node Jobs

The previous example was f

[openstack-dev] Zuul v3 - What's Coming: What to expect with the Zuul v3 Rollout

[Monty Taylor]

Hi everybody!

This content can also be found at
http://inaugust.com/posts/whats-coming-zuulv3.html - but I've pasted it
in here directly because I know that some folks don't like clicking links.

tl;dr - At last week's OpenStack PTG, the OpenStack Infra team ran the
first Zuul v3 job, so it's time to start getting everybody ready for
what's coming

**Don't Panic!** Awesome changes are coming, but you are NOT on the hook
for rewriting all of your project's gate jobs or anything crazy like
that. Now grab a seat by the fire, pour yourself a drink while I spin a
yarn about days gone by and days yet to come.

First, some background

The OpenStack Infra team has been hard at work for quite a while on a
new version of Zuul (where by 'quite some time' I mean that Jim Blair
and I had our first Zuul v3 design whiteboarding session in 2014). As
you might be able to guess given the amount of time, there are some big
things coming that will have a real and visible impact on the OpenStack
community and beyond. Since we have a running Zuul v3 now [1], it seemed
like the time to start getting folks up to speed on what to expect.

There is other deep-dive information on architecture and rationale if
you're interested[2], but for now we'll focus on what's relevant for end
users. We're also going to start sending out a bi-weekly "Status of Zuul
v3" email to the openstack-dev@lists.openstack.org mailing list ... so
stay tuned!

**Important Note** This post includes some code snippets - but v3 is
still a work in progress. We know of at least one breaking change that
is coming to the config format, so please treat this not as a tutorial,
but as a conceptual overview. Syntax is subject to change.

The Big Ticket Items

While there are a bunch of changes behind the scenes, there are a
reasonably tractable number of user-facing differences.

* Self-testing In-Repo Job Config
* Ansible Job Content
* First-class Multi-node Jobs
* Improved Job Reuse
* Support for non-OpenStack Code and Node Systems
* and Much, Much More

Self-testing In-Repo Job Config

This is probably the biggest deal. There are a lot of OpenStack Devs
(around 2k in Ocata) and a lot of repositories (1689) There a lot fewer
folks on the project-config-core team who are the ones who review all of
the job config changes (please everyone thank Andreas Jaeger next time
you see him). That's not awesome.

Self-testing in-repo job config is awesome.

Many systems out there these days have an in-repo job config system.
Travis CI has had it since day one, and Jenkins has recently added
support for a Jenkinsfile inside of git repos. With Zuul v3, we'll have
it too.

Once we roll out v3 to everyone, as a supplement to jobs defined in our
central config repositories, each project will be able to add a
zuul.yaml file to their own repo:


- job:
name: my_awesome_job
nodes:
  - name: controller
label: centos-7

- project:
name: openstack/awesome_project
check:
  jobs:
- my_awesome_job

It's a small file, but there is a lot going on, so let's unpack it.

First we define a job to run. It's named my_awesome_job and it needs one
node. That node will be named controller and will be based on the
centos-7 base node in nodepool.

In the next section, we say that we want to run that job in the check
pipeline, which in OpenStack is defined as the jobs that run when
patchsets are proposed.

And it's also self-testing!

Everyone knows the fun game of writing a patch to the test jobs, getting
it approved, then hoping it works once it starts running. With Zuul v3
in-repo jobs, if there is a change to job definitions in a proposed
patch, that patch will be tested with those changes applied. And since
it's Zuul, Depends-On footers are honored as well - so iteration on
getting a test job right becomes just like iterating on any other patch
or sequence of patches.

Ansible Job Content

The job my_awesome_job isn't very useful if it doesn't define any
content. That's done in the repo as well, in playbooks/my_awesome_job.yaml:


- hosts: controller
  tasks:
- name: Run make tests
  shell: make distcheck

As previously mentioned, the job content is now defined in Ansible
rather than using our Jenkins Job Builder tool. This playbook is going
to run a tasks on a host called controller which you may remember we
requested in the job definition. On that host, it will run make
distcheck. Pretty much anything you can do in Ansible, you can do in a
Zuul job now, and the playbooks should also be re-usable outside of a
testing context.

First Class Multi-Node Jobs

The previous example was for running a job on a node. What if you want
to do multi-node?


- job:
name: my_awesome_job
nodes:
  - name: controller
label: ubuntu-xenial
  - name: compute
label: centos-7

- project:
name: openstack/awesome_project
check:
  jobs:
- my_awesome_job

As you may have surmised, nodes is a list, so you can have more than
one.