Re: [OpenStack-Infra] Some branch issues with Zuul

[James E. Blair]

Doug Hellmann  writes:

> In the discussion yesterday, and in the emails today, you've implied
> that there is an ordering to job definitions beyond inheritance. Is that
> discovery order documented somewhere? If not, is it simple enough to
> describe in a few sentences here? Are repositories scanned in a
> particular order, for example? Or is it based on something else?

There's some discussion of it here:

https://docs.openstack.org/infra/zuul/feature/zuulv3/user/config.html#job

  When Zuul decides to run a job, it performs a process known as
  freezing the job. Because any number of job variants may be
  applicable, Zuul collects all of the matching variants and applies
  them in the order they appeared in the configuration. The resulting
  frozen job is built from attributes gathered from all of the matching
  variants. In this way, exactly what is run is dependent on the
  pipeline, project, branch, and content of the item.

Because top-level job variants may only be defined in the same project
(so that one project may not alter the jobs defined by another project),
the order that the repos are loaded doesn't matter for this; only the
order that branches are loaded within a repo.  That's not specified by
the documentation, though it is currently 'master' followed by others in
alphabetical order.

The proposed changes would reduce the importance of that, since master
will have an implied branch matcher, meaning that by default, jobs in
master won't have an effect on other branches.  I'd probably still leave
the order the same though, in case someone wanted to override that
behavior.

In practice, and especially with the proposed change to have an implied
branch matcher on master, the ordering aspect is most likely to be
visible when a user adds several variants of a job in the same file.

The order that the repos themselves are loaded is important, however, in
inheritance.  They are loaded in a defined order (the order they appear
in the main.yaml tenant configuration file), and currently, a job may
only inherit from another job which has already been defined.  So we
manually put more "general-use" projects (e.g., devstack, tempest)
earlier in the config.  I would characterize this as an oversight, and
was planning on fixing it soon regardless, however, the proposal to
perform late-binding inheritance will solve it as well (since the
inheritance path would be determined when the job is run, well after all
the configuration is loaded).

There's some more discussion of the repo loading order here:

  
https://docs.openstack.org/infra/zuul/feature/zuulv3/admin/tenants.html#attr-tenant

-Jim

___
OpenStack-Infra mailing list
OpenStack-Infra@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra

Re: [OpenStack-Infra] Some branch issues with Zuul

[James E. Blair]

Andrea Frittoli  writes:

> We will need somewhere in the logs a description of the traversal that was
> done to build the final job. I believe that would help debugging issues that
> may arise from unexpected inheritance behaviour.
>
> Andrea Frittoli (andreaf)

Yes!  We have that to some degree today, and it's been very helpful
lately. :)

Example:

  
http://logs.openstack.org/33/509233/7/check/tox-linters/23b39f2/zuul-info/inventory.yaml

  _inheritance_path:
  - 'inherit from '
  - 'inherit from '
  - 'inherit from '
  - 'self '
  - 'apply variant '

The var name has an underscore because it's undocumented and (very!)
likely to change.  It's basically just a some debug lines rendered as a
list at the moment.  Once it settles down, I plan on turning it into a
proper data structure and documenting it.  Then it may be appropriate to
output it in the console log.

In the late-binding proposal, the distinction between inheritance and
variance is largely lost, and it essentially just turns into a list of
'apply variant' lines.  I think even that would still be enough,
However, I may be able to improve the output slightly and get a bit more
context back.  I'll look at that when I make the final version of the
patch.

-Jim

___
OpenStack-Infra mailing list
OpenStack-Infra@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra

Re: [OpenStack-Infra] Some branch issues with Zuul

[Doug Hellmann]

Excerpts from corvus's message of 2017-10-24 15:22:51 -0700:
> Hi,
> 
> A number of issues related to how jobs are defined and run on projects
> with stable branches have come up recently.  I believe they are all
> related, and they, as well as the solutions to them, must be considered
> together.
> 
> 
> The Problems
> 
> 
> I've identified the following five problems:
> 
> 1) Parents should have variants applied
> 
> Currently inheritance and variance are distinct.  Inheritance modifies a
> job's configuration statically at the time the configuration is loaded.
> Variants are applied dynamically right before the job runs.
> 
> That means that when a job starts, we pick a single job to start with,
> then apply variants of that particular job.  But since the inheritance
> has already been applied, any variants which a user may expect to apply
> to a parent job will not be applied.  When the inheritance chain is
> created at configuration time, only the "reference" definition of the
> job is used -- that is, the first appearance of the job.
> 
> In other words, more than likely, most jobs defined in a stable branch
> are going to inherit from a parent job defined on the master branch --
> even if that parent job has a stable branch variant.
> 
> 2) Variants may cause parents to be ignored
> 
> We currently ignore the parent attribute on a job variant.  If we did
> not, then when the variant is applied, it would pull in all of the
> attributes of its parent, which is likely to be on the master branch
> (since its parent will, as in #1, be the reference definition).
> 
> This ignoring of the parent attribute happens at configuration time
> (naturally, since that is when inheritance is applied).
> 
> This means that if the first job that matches a change is a variant
> (i.e., the reference definition of a job has a non-matching branch
> matcher), this top-level variant job will not actually have a parent.

In the discussion yesterday, and in the emails today, you've implied
that there is an ordering to job definitions beyond inheritance. Is that
discovery order documented somewhere? If not, is it simple enough to
describe in a few sentences here? Are repositories scanned in a
particular order, for example? Or is it based on something else?

Doug

> 
> 3) Variants may add duplicate pre/post playbooks
> 
> Currently, the master branch does not have an implied branch matcher, so
> jobs that exist on master and stable branches will generally be derived
> from both.
> 
> If such a job adds a pre or post playbook, the job that is ultimately
> created and run for a change on the stable branch may have those added
> by both the variant defined on the master branch as well as that defined
> on the stable branch (since pre and post playbooks are cumulative).
> 
> 4) Variants on branches without explicit playbooks should use branch
>playbooks
> 
> Whenever a job includes a pre-run, run (including the implicit run), or
> post-run playbook, Zuul remembers where and uses that branch of that
> repo to run that playbook.  If a job were constructed from the master
> branch, and then had a stable branch variant applied but did not repeat
> the pre-run, run, or post-run attributes from the master, then Zuul
> would end up attempting to run the playbook from the master branch
> rather than the stable.
> 
> 5) The master branch should have implied branch matchers
> 
> Currently jobs defined in an untrusted project on any branch other than
> 'master' have an implicit branch matcher applied to them.  This is what
> allows the version of a job in a stable branch to only affect the stable
> branch.  The fact that there is no implicit branch matcher applied to
> the master branch is what allows jobs defined in zuul-jobs to run on
> changes to any branch.
> 
> However, this also means that jobs on stable branches are frequently
> built from variants on both the master and stable branch.  This may work
> for a short while, but will fail as soon as someone wants to add
> something to the master branch which should not exist on the stable
> branch (e.g., enabling a new service by default).
> 
> 
> The Design Considerations
> =
> 
> In looking at these, I believe they have come about because of two
> design goals which we did not appreciate were in moderate tension with
> each other:
> 
> A) In-repo configuration for a project should apply to that branch.
> Changing the behavior of a job on a stable branch should merely involve
> changing the configuration or playbook in that stable branch.  When a
> project branches master to create a new stable branch, both branches
> should initially have the same content and behavior, but then evolve
> independently.
> 
> B) We should be able to define jobs not only in a central repository
> such as project-config, but a central *untrusted* repository such as
> zuul-jobs or openstack-zuul-jobs.
> 
> I think these are valid and important to keep in mind as we 

Re: [OpenStack-Infra] Some guidance on job variants

[Andreas Jaeger]

On 2017-10-25 00:51, James E. Blair wrote:
> At the infra meeting today, we discussed how to handle job variants.  I
> will try to summarize the discussion and extrapolate some things.
> 
> The Zuul v3 migration doc in the infra manual is very clear that some
> project-templates should only be added in project-config, rather than
> in-repo[1].
> 
> However, there are some edge cases worth considering:
> 
> First: projects may want to alter the behavior of those jobs (e.g., to
> run unit tests on an older node type).  In these cases projects should
> leave the template in project-config, and just alter the job by adding a
> project-pipeline variant in-repo.
> 
> Theoretically, projects could use this to make their py27 jobs
> non-voting.  That would be a TC policy violation, but this system is not
> designed to enforce that policy, only facilitate it.
> 
> Second: projects may want to cause those jobs not to run in some
> circumstances.
> 
> Zuul's configuration for job variants is additive.  Any job variants
> which match a change will be cumulatively applied to the final job
> configuration before it runs.  However, once any job or variant for a
> project-pipeline matches a change, that job will run.  There is not a
> way to have one variant match a change, and then have a second also
> match it and somehow cause the job not to run.  Variants which don't
> match the job simply don't add their own attributes to it.
> 
> This means that if a project wants to alter the files or irrelevant
> files list for a job covered by one of those templates, or avoid running
> a job on a particular branch, there is no way to do that if the template
> is applied in project-config.
> 
> In these cases, I think the following policy should apply:
> 
> 1) If you can use the project-template as-is, then it should be applied
> to the project in project-config.
> 
> 2) If you want to improve the files or irrelevant-files matchers on the
> template in such a way that they can apply safely to everyone using the
> template, please do so.
> 
> 3) If your project needs a job variant that is incompatible with the
> template, then remove it from project-config and add the individual jobs
> to the project in-repo.  The project is still responsible for adhering
> to the PTI.
> 
> As you can see, these aren't hard and fast rules, this is more of an
> attempt to gain clarity and be able to make helpful and consistent
> suggestions to folks about how to configure jobs in certain ways.


I'm fine with this and suggest to update the infra manual with the
policy. Also, the explanation about job variants should be in the infra
manual,

Andreas
-- 
 Andreas Jaeger aj@{suse.com,opensuse.org} Twitter: jaegerandi
  SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany
   GF: Felix Imendörffer, Jane Smithard, Graham Norton,
   HRB 21284 (AG Nürnberg)
GPG fingerprint = 93A3 365E CE47 B889 DF7F  FED1 389A 563C C272 A126


___
OpenStack-Infra mailing list
OpenStack-Infra@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra

Re: [OpenStack-Infra] Some branch issues with Zuul

[Andrea Frittoli]

On Wed, Oct 25, 2017 at 12:23 AM James E. Blair  wrote:

> Hi,
>
> A number of issues related to how jobs are defined and run on projects
> with stable branches have come up recently.  I believe they are all
> related, and they, as well as the solutions to them, must be considered
> together.
>
>
> The Problems
> 
>
> I've identified the following five problems:
>
> 1) Parents should have variants applied
>
> Currently inheritance and variance are distinct.  Inheritance modifies a
> job's configuration statically at the time the configuration is loaded.
> Variants are applied dynamically right before the job runs.
>
> That means that when a job starts, we pick a single job to start with,
> then apply variants of that particular job.  But since the inheritance
> has already been applied, any variants which a user may expect to apply
> to a parent job will not be applied.  When the inheritance chain is
> created at configuration time, only the "reference" definition of the
> job is used -- that is, the first appearance of the job.
>
> In other words, more than likely, most jobs defined in a stable branch
> are going to inherit from a parent job defined on the master branch --
> even if that parent job has a stable branch variant.
>
> 2) Variants may cause parents to be ignored
>
> We currently ignore the parent attribute on a job variant.  If we did
> not, then when the variant is applied, it would pull in all of the
> attributes of its parent, which is likely to be on the master branch
> (since its parent will, as in #1, be the reference definition).
>
> This ignoring of the parent attribute happens at configuration time
> (naturally, since that is when inheritance is applied).
>
> This means that if the first job that matches a change is a variant
> (i.e., the reference definition of a job has a non-matching branch
> matcher), this top-level variant job will not actually have a parent.
>
> 3) Variants may add duplicate pre/post playbooks
>
> Currently, the master branch does not have an implied branch matcher, so
> jobs that exist on master and stable branches will generally be derived
> from both.
>
> If such a job adds a pre or post playbook, the job that is ultimately
> created and run for a change on the stable branch may have those added
> by both the variant defined on the master branch as well as that defined
> on the stable branch (since pre and post playbooks are cumulative).
>
> 4) Variants on branches without explicit playbooks should use branch
>playbooks
>
> Whenever a job includes a pre-run, run (including the implicit run), or
> post-run playbook, Zuul remembers where and uses that branch of that
> repo to run that playbook.  If a job were constructed from the master
> branch, and then had a stable branch variant applied but did not repeat
> the pre-run, run, or post-run attributes from the master, then Zuul
> would end up attempting to run the playbook from the master branch
> rather than the stable.
>
> 5) The master branch should have implied branch matchers
>
> Currently jobs defined in an untrusted project on any branch other than
> 'master' have an implicit branch matcher applied to them.  This is what
> allows the version of a job in a stable branch to only affect the stable
> branch.  The fact that there is no implicit branch matcher applied to
> the master branch is what allows jobs defined in zuul-jobs to run on
> changes to any branch.
>
> However, this also means that jobs on stable branches are frequently
> built from variants on both the master and stable branch.  This may work
> for a short while, but will fail as soon as someone wants to add
> something to the master branch which should not exist on the stable
> branch (e.g., enabling a new service by default).
>
>
> The Design Considerations
> =
>
> In looking at these, I believe they have come about because of two
> design goals which we did not appreciate were in moderate tension with
> each other:
>
> A) In-repo configuration for a project should apply to that branch.
> Changing the behavior of a job on a stable branch should merely involve
> changing the configuration or playbook in that stable branch.  When a
> project branches master to create a new stable branch, both branches
> should initially have the same content and behavior, but then evolve
> independently.
>
> B) We should be able to define jobs not only in a central repository
> such as project-config, but a central *untrusted* repository such as
> zuul-jobs or openstack-zuul-jobs.
>
> I think these are valid and important to keep in mind as we consider
> solutions.
>
>
> The Changes
> ===
>
> I believe the following changes will address all five problems and
> achieve both design goals:
>
> a) Apply inheritance at the same time as variance
>
> Rather than applying inheritance at configuration time, apply it at the
> time the job is frozen before being run.  We can perform a depth-first
>