On Sat, Jan 27, 2018 at 2:57 AM, James E. Blair <cor...@inaugust.com> wrote: > Balázs Gibizer <balazs.gibi...@ericsson.com> writes: > >> Hi, >> >> I'm getting more and more confused how the zuul job hierarchy works or >> is supposed to work. > > Hi! > > First, you (or others) may or may not have seen this already -- some of > it didn't exist when we first rolled out v3, and some of it has changed > -- but here are the relevant bits of the documentation that should help > explain what's going on. It helps to understand freezing: > > https://docs.openstack.org/infra/zuul/user/config.html#job > > and matching: > > https://docs.openstack.org/infra/zuul/user/config.html#matchers > >> First there was a bug in nova that some functional tests are not >> triggered although the job (re-)definition in the nova part of the >> project-config should not prevent it to run [1]. >> >> There we figured out that irrelevant-files parameter of the jobs are >> not something that can be overriden during re-definition or through >> parent-child relationship. The base job openstack-tox-functional has >> an irrelevant-files attribute that lists '^doc/.*$' as a path to be >> ignored [2]. In the other hand the nova part of the project-config >> tries to make this ignore less broad by adding only '^doc/source/.*$' >> . This does not work as we expected and the job did not run on changes >> that only affected ./doc/notification_samples path. We are fixing it >> by defining our own functional job in nova tree [4]. >> >> [1] https://bugs.launchpad.net/nova/+bug/1742962 >> [2] >> https://github.com/openstack-infra/openstack-zuul-jobs/blob/1823e3ea20e6dfaf37786a6ff79c56cb786bf12c/zuul.d/jobs.yaml#L380 >> [3] >> https://github.com/openstack-infra/project-config/blob/1145ab1293f5fa4d34c026856403c22b091e673c/zuul.d/projects.yaml#L10509 >> [4] https://review.openstack.org/#/c/533210/ > > This is correct. The issue here is that the irrelevant-files definition > on openstack-tox-functional is too broad. We need to be *extremely* > careful applying matchers to jobs like that. Generally I think that > irrelevant-files should be reserved for the project-pipeline invocations > only. That's how they were effectively used in Zuul v2, after all. > > Essentially, when someone puts an irrelevant-files section on a job like > that, they are saying "this job will never apply to these files, ever." > That's clearly not correct in this case. > > So our solutions are to acknowledge that it's over-broad, and reduce or > eliminate the list in [2] and expand it elsewhere (as in [3]). Or we > can say "we were generally correct, but nova is extra special so it > needs its own job". If that's the choice, then I think [4] is a fine > solution. > >> Then I started looking into other jobs to see if we made similar >> mistakes. I found two other examples in the nova related jobs where >> redefining the irrelevant-files of a job caused problems. In these >> examples nova tried to ignore more paths during the override than what >> was originally ignored in the job definition but that did not work >> [5][6]. >> >> [5] https://bugs.launchpad.net/nova/+bug/1745405 (temptest-full) > > As noted in that bug, the tempest-full job is invoked on nova via this > stanza: > > https://github.com/openstack-infra/project-config/blob/5ddbd62a46e17dd2fdee07bec32aa65e3b637ff3/zuul.d/projects.yaml#L10674-L10688 > > As expected, that did not match. There is a second invocation of > tempest-full on nova here: > > http://git.openstack.org/cgit/openstack-infra/openstack-zuul-jobs/tree/zuul.d/zuul-legacy-project-templates.yaml#n126 > > That has no irrelevant-files matches, and so matches everything. If you > drop the use of that template, it will work as expected. Or, if you can > say with some certainty that nova's irrelevant-files set is not > over-broad, you could move the irrelevant-files from nova's invocation > into the template, or even the job, and drop nova's individual > invocation. > >> [6] https://bugs.launchpad.net/nova/+bug/1745431 (neutron-grenade) > > The same template invokes this job as well. > >> So far the problem seemed to be consistent (i.e. override does not >> work). But then I looked into neutron-grenade-multinode. That job is >> defined in neutron tree (like neutron-grenade) but nova also refers to >> it in nova section of the project-config with different >> irrelevant-files than their original definition. So I assumed that >> this will lead to similar problem than in case of neutron-grenade, but >> it doesn't. >> >> The neutron-grenade-multinode original definition [1] does not try to >> ignore the 'nova/tests' path but the nova side of the definition in >> the project config does try to ignore that path [8]. Interestingly a >> patch in nova that only changes under the path: nova/tests/ does not >> trigger the job [9]. So in this case overriding the irrelevant-files >> of a job works. (It seems that overriding neutron-tempest-linuxbridge >> irrelevant-files works too). >> >> [7] >> https://github.com/openstack/neutron/blob/7e3d6a18fb928bcd303a44c1736d0d6ca9c7f0ab/.zuul.yaml#L140-L159 >> [8] >> https://github.com/openstack-infra/project-config/blob/5ddbd62a46e17dd2fdee07bec32aa65e3b637ff3/zuul.d/projects.yaml#L10516-L10530 >> [9] https://review.openstack.org/#/c/537936/ >> >> I don't see what is the difference between neutron-grenade and >> neutron-grenade-multinode jobs definitions from this perspective but >> it seems that the irrelevent-files attribute behaves inconsistently >> in these two jobs. Could you please help me undestand how >> irrelevant-files in overriden jobs supposed to work? > > These jobs only have the one invocation -- on the nova project -- and > are not added via a template. > > Hopefully that explains the difference. > > Basically, the irrelevant-files on at least one project-pipeline > invocation of a job have to match, as well as at least one definition of > the job. So if both things have irrelevant-files, then it's effectively > a union of the two.
Thanks a lot for clarifying those bits. I thought irrelevant-files are always overridden not append/union as they are regular expression or list of regular expressions. It is clear now that where irrelevant-files are present i both job definition and project-pipeline invocation job list then it is union of these two. Is it same for inherited job case also or it is completely overridden? I mean if base job has irrelevant-files and then inherited job from this base job define irrelevant-files then, is it union or overridden. One more question, as mentioned in doc that 'irrelevant-files' and 'files' should not be defined together in single job. what happen if it is defined? as base job usually have 'irrelevant-files' and if any inherited job defined 'files' then, is it error or unexpected behaviour ? Because answer to above query convey whether we should declare the 'irrelevant-files', 'files' in job definition or not ? > > I used a tool to help verify some of the information in this message, > especially the bugs [5] and [6]. You can ask Zuul to output debug > information about its job selection if you're dealing with confusing > situations like this. I went ahead and pushed a new patchset to your > test change to demonstrate how: > > https://review.openstack.org/537936 > > When it finishes running all the tests (in a few hours), it should > include in its report debug information about the decision-making > process for the jobs it ran. It outputs similar information into the > debug logs; so that we don't have to wait for it to see what it looks > like here is that copy: > > http://paste.openstack.org/show/653729/ > > The relevant lines for [5] are: > > 2018-01-26 13:07:53,560 DEBUG zuul.layout: Pipeline variant <Job tempest-full > branches: None source: > openstack-infra/openstack-zuul-jobs/zuul.d/zuul-legacy-project-templates.yaml@master#126> > matched <Change 0x7f2fed8883c8 537936,2> > 2018-01-26 13:07:53,560 DEBUG zuul.layout: Pipeline variant <Job tempest-full > branches: None source: > openstack-infra/project-config/zuul.d/projects.yaml@master#10485> did not > match <Change 0x7f2fed8883c8 537936,2> > > Note the project-file-branch-line-number references are especially > helpful. > > -Jim > > __________________________________________________________________________ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -gmann __________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev