Re: [openstack-dev] [nova][neutron][infra] zuul job definitions overrides and the irrelevant-file attribute
On Thu, Feb 15, 2018 at 1:43 AM, Andrea Frittoli wrote: > > > On Wed, Feb 14, 2018 at 4:05 PM James E. Blair wrote: >> >> Andrea Frittoli writes: >> >> >> That has no irrelevant-files matches, and so matches everything. If >> >> you >> >> drop the use of that template, it will work as expected. Or, if you >> >> can >> >> say with some certainty that nova's irrelevant-files set is not >> >> over-broad, you could move the irrelevant-files from nova's invocation >> >> into the template, or even the job, and drop nova's individual >> >> invocation. >> >> >> > I don't think projects in the integrated gate should remove themselves >> > from the >> > template, it really helps keeping consistency. >> > >> > The pattern I've seen is that most projects repeat the same list of >> > irrelevant files >> > over and over again in all of their integration tests, It would be handy >> > in >> > future to >> > be able to set irrelevant-files on a template when it's consumed. >> > So we could have shared irrelevant files defined in the template, and >> > custom ones >> > added by each project when consuming the template. I don't this is is >> > possible today. >> > Does it sound like a reasonable feature request? >> >> A template may specify many jobs, so if we added something like that >> feature, what would the project-pipeline template application's >> irrelevant files apply to? All of the jobs in the template? We could >> do that. > > > That's what I was thinking about, > >> >> But it only takes one exception for this approach to fall >> short, and while a lot of irrelevant-files stanzas for a project are >> similar, I don't think having exceptions will be unusual. The only way >> to handle exceptions like that is to specify them with jobs, and we're >> back in the same situation. >> >> Also, combining irrelevant-files is very difficult to think about. >> Because it's two inverse matches, combining them ends up being the >> intersection, not the union. So if we implemented this, I don't think >> we should have any irrelevant-files in the template, only on template >> application. >> >> I still tend to think that irrelevant-files are almost invariably >> project-specific, so we should avoid using them in templates and job >> definitions (unless absolutely certain they are universally applicable), >> and we should only define them in one place -- in the project-pipeline >> definition for individual jobs. > > > I agree with your concerns, but the problem is that the current > implementation > renders job templates rather useless. Each project has to re-add each job in > a > template in its pipeline content definition to be able to apply irrelevant > files, and > that will turn stale if a template is modified. > > With the migration to zuulv3 native jobs there is a lot of job renaming and > adding/ > removing jobs going on, so for instance if a job is removed what used to be > a setting > irrelevant files may become running an extra job. > > I don't really have a solution for this, but perhaps someone has an idea? I am in favor of idea of not defining the irrelevant_files in base job definition or template and have them defined by project in project-pipeline only. This solve most of the issue even that can ask each project to define the common irrelevant_files. With that, should we keep the Template limited to system one only which are mentioned here [1] ? i mean no 'integrate-gate' etc template. ..1 https://docs.openstack.org/infra/manual/zuulv3.html#what-to-convert -gmann > > Andrea Frittoli (andreaf) > >> >> -Jim > > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova][neutron][infra] zuul job definitions overrides and the irrelevant-file attribute
On Wed, Feb 14, 2018 at 4:05 PM James E. Blair wrote: > Andrea Frittoli writes: > > >> That has no irrelevant-files matches, and so matches everything. If you > >> drop the use of that template, it will work as expected. Or, if you can > >> say with some certainty that nova's irrelevant-files set is not > >> over-broad, you could move the irrelevant-files from nova's invocation > >> into the template, or even the job, and drop nova's individual > >> invocation. > >> > > I don't think projects in the integrated gate should remove themselves > > from the > > template, it really helps keeping consistency. > > > > The pattern I've seen is that most projects repeat the same list of > > irrelevant files > > over and over again in all of their integration tests, It would be handy > in > > future to > > be able to set irrelevant-files on a template when it's consumed. > > So we could have shared irrelevant files defined in the template, and > > custom ones > > added by each project when consuming the template. I don't this is is > > possible today. > > Does it sound like a reasonable feature request? > > A template may specify many jobs, so if we added something like that > feature, what would the project-pipeline template application's > irrelevant files apply to? All of the jobs in the template? We could > do that. That's what I was thinking about, > But it only takes one exception for this approach to fall > short, and while a lot of irrelevant-files stanzas for a project are > similar, I don't think having exceptions will be unusual. The only way > to handle exceptions like that is to specify them with jobs, and we're > back in the same situation. > > Also, combining irrelevant-files is very difficult to think about. > Because it's two inverse matches, combining them ends up being the > intersection, not the union. So if we implemented this, I don't think > we should have any irrelevant-files in the template, only on template > application. > > I still tend to think that irrelevant-files are almost invariably > project-specific, so we should avoid using them in templates and job > definitions (unless absolutely certain they are universally applicable), > and we should only define them in one place -- in the project-pipeline > definition for individual jobs. > I agree with your concerns, but the problem is that the current implementation renders job templates rather useless. Each project has to re-add each job in a template in its pipeline content definition to be able to apply irrelevant files, and that will turn stale if a template is modified. With the migration to zuulv3 native jobs there is a lot of job renaming and adding/ removing jobs going on, so for instance if a job is removed what used to be a setting irrelevant files may become running an extra job. I don't really have a solution for this, but perhaps someone has an idea? Andrea Frittoli (andreaf) > -Jim > __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova][neutron][infra] zuul job definitions overrides and the irrelevant-file attribute
Andrea Frittoli writes: >> That has no irrelevant-files matches, and so matches everything. If you >> drop the use of that template, it will work as expected. Or, if you can >> say with some certainty that nova's irrelevant-files set is not >> over-broad, you could move the irrelevant-files from nova's invocation >> into the template, or even the job, and drop nova's individual >> invocation. >> > I don't think projects in the integrated gate should remove themselves > from the > template, it really helps keeping consistency. > > The pattern I've seen is that most projects repeat the same list of > irrelevant files > over and over again in all of their integration tests, It would be handy in > future to > be able to set irrelevant-files on a template when it's consumed. > So we could have shared irrelevant files defined in the template, and > custom ones > added by each project when consuming the template. I don't this is is > possible today. > Does it sound like a reasonable feature request? A template may specify many jobs, so if we added something like that feature, what would the project-pipeline template application's irrelevant files apply to? All of the jobs in the template? We could do that. But it only takes one exception for this approach to fall short, and while a lot of irrelevant-files stanzas for a project are similar, I don't think having exceptions will be unusual. The only way to handle exceptions like that is to specify them with jobs, and we're back in the same situation. Also, combining irrelevant-files is very difficult to think about. Because it's two inverse matches, combining them ends up being the intersection, not the union. So if we implemented this, I don't think we should have any irrelevant-files in the template, only on template application. I still tend to think that irrelevant-files are almost invariably project-specific, so we should avoid using them in templates and job definitions (unless absolutely certain they are universally applicable), and we should only define them in one place -- in the project-pipeline definition for individual jobs. -Jim __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova][neutron][infra] zuul job definitions overrides and the irrelevant-file attribute
On Fri, Jan 26, 2018 at 5:57 PM James E. Blair wrote: > Balázs Gibizer writes: > > > Hi, > > > > I'm getting more and more confused how the zuul job hierarchy works or > > is supposed to work. > > Hi! > > First, you (or others) may or may not have seen this already -- some of > it didn't exist when we first rolled out v3, and some of it has changed > -- but here are the relevant bits of the documentation that should help > explain what's going on. It helps to understand freezing: > > https://docs.openstack.org/infra/zuul/user/config.html#job > > and matching: > > https://docs.openstack.org/infra/zuul/user/config.html#matchers > > > First there was a bug in nova that some functional tests are not > > triggered although the job (re-)definition in the nova part of the > > project-config should not prevent it to run [1]. > > > > There we figured out that irrelevant-files parameter of the jobs are > > not something that can be overriden during re-definition or through > > parent-child relationship. The base job openstack-tox-functional has > > an irrelevant-files attribute that lists '^doc/.*$' as a path to be > > ignored [2]. In the other hand the nova part of the project-config > > tries to make this ignore less broad by adding only '^doc/source/.*$' > > . This does not work as we expected and the job did not run on changes > > that only affected ./doc/notification_samples path. We are fixing it > > by defining our own functional job in nova tree [4]. > > > > [1] https://bugs.launchpad.net/nova/+bug/1742962 > > [2] > > > https://github.com/openstack-infra/openstack-zuul-jobs/blob/1823e3ea20e6dfaf37786a6ff79c56cb786bf12c/zuul.d/jobs.yaml#L380 > > [3] > > > https://github.com/openstack-infra/project-config/blob/1145ab1293f5fa4d34c026856403c22b091e673c/zuul.d/projects.yaml#L10509 > > [4] https://review.openstack.org/#/c/533210/ > > This is correct. The issue here is that the irrelevant-files definition > on openstack-tox-functional is too broad. We need to be *extremely* > careful applying matchers to jobs like that. Generally I think that > irrelevant-files should be reserved for the project-pipeline invocations > only. That's how they were effectively used in Zuul v2, after all. > > Essentially, when someone puts an irrelevant-files section on a job like > that, they are saying "this job will never apply to these files, ever." > That's clearly not correct in this case. > > So our solutions are to acknowledge that it's over-broad, and reduce or > eliminate the list in [2] and expand it elsewhere (as in [3]). Or we > can say "we were generally correct, but nova is extra special so it > needs its own job". If that's the choice, then I think [4] is a fine > solution. > > > Then I started looking into other jobs to see if we made similar > > mistakes. I found two other examples in the nova related jobs where > > redefining the irrelevant-files of a job caused problems. In these > > examples nova tried to ignore more paths during the override than what > > was originally ignored in the job definition but that did not work > > [5][6]. > > > > [5] https://bugs.launchpad.net/nova/+bug/1745405 (temptest-full) > > As noted in that bug, the tempest-full job is invoked on nova via this > stanza: > > > https://github.com/openstack-infra/project-config/blob/5ddbd62a46e17dd2fdee07bec32aa65e3b637ff3/zuul.d/projects.yaml#L10674-L10688 > > As expected, that did not match. There is a second invocation of > tempest-full on nova here: > > > http://git.openstack.org/cgit/openstack-infra/openstack-zuul-jobs/tree/zuul.d/zuul-legacy-project-templates.yaml#n126 I guess the line number changed since so this has moved to L101 [1] now :). tempest-full is part of the integrated-gate, so all projects in it run it through there. [1] http://git.openstack.org/cgit/openstack-infra/openstack-zuul-jobs/tree/zuul.d/zuul-legacy-project-templates.yaml#n101 > > > That has no irrelevant-files matches, and so matches everything. If you > drop the use of that template, it will work as expected. Or, if you can > say with some certainty that nova's irrelevant-files set is not > over-broad, you could move the irrelevant-files from nova's invocation > into the template, or even the job, and drop nova's individual > invocation. > > I don't think projects in the integrated gate should remove themselves from the template, it really helps keeping consistency. The pattern I've seen is that most projects repeat the same list of irrelevant files over and over again in all of their integration tests, It would be handy in future to be able to set irrelevant-files on a template when it's consumed. So we could have shared irrelevant files defined in the template, and custom ones added by each project when consuming the template. I don't this is is possible today. Does it sound like a reasonable feature request? Andrea Frittoli (andreaf) > > [6] https://bugs.launchpad.net/nova/+bug/1745431 (neutron-grenade) > > The same template invokes this jo
Re: [openstack-dev] [nova][neutron][infra] zuul job definitions overrides and the irrelevant-file attribute
On Fri, Jan 26, 2018 at 2:05 PM Balázs Gibizer wrote: > Hi, > > I'm getting more and more confused how the zuul job hierarchy works or is > supposed to work. > > First there was a bug in nova that some functional tests are not triggered > although the job (re-)definition in the nova part of the project-config > should not prevent it to run [1]. > > There we figured out that irrelevant-files parameter of the jobs are not > something that can be overriden during re-definition or through > parent-child relationship. The base job openstack-tox-functional has an > irrelevant-files attribute that lists '^doc/.*$' as a path to be ignored > [2]. In the other hand the nova part of the project-config tries to make > this ignore less broad by adding only '^doc/source/.*$' . This does not > work as we expected and the job did not run on changes that only affected > ./doc/notification_samples path. We are fixing it by defining our own > functional job in nova tree [4]. > > [1] https://bugs.launchpad.net/nova/+bug/1742962 > [2] > https://github.com/openstack-infra/openstack-zuul-jobs/blob/1823e3ea20e6dfaf37786a6ff79c56cb786bf12c/zuul.d/jobs.yaml#L380 > [3] > https://github.com/openstack-infra/project-config/blob/1145ab1293f5fa4d34c026856403c22b091e673c/zuul.d/projects.yaml#L10509 > [4] https://review.openstack.org/#/c/533210/ > > Then I started looking into other jobs to see if we made similar mistakes. > I found two other examples in the nova related jobs where redefining the > irrelevant-files of a job caused problems. In these examples nova tried to > ignore more paths during the override than what was originally ignored in > the job definition but that did not work [5][6]. > > [5] https://bugs.launchpad.net/nova/+bug/1745405 (temptest-full) > [6] https://bugs.launchpad.net/nova/+bug/1745431 (neutron-grenade) > > So far the problem seemed to be consistent (i.e. override does not work). > But then I looked into neutron-grenade-multinode. That job is defined in > neutron tree (like neutron-grenade) > That is wrong and it should not have happened. Grenade jobs that are shared by all the repos in the integrated gate should live in repos owned by QA/infra - it was never the plan for them to end up in the neutron repo. We're working on making grenade and multinode zuulv3 native jobs. Grenade jobs will leave in the grenade repo, once they're ready we will remove the legacy one from neutron side and add the new ones defined in grenade. Changes will be done to the job template accordingly which means that for teams that are consuming those jobs via the template only there'll be no action. Andrea Frittoli (andreaf) > but nova also refers to it in nova section of the project-config with > different irrelevant-files than their original definition. So I assumed > that this will lead to similar problem than in case of neutron-grenade, but > it doesn't. > > The neutron-grenade-multinode original definition [1] does not try to > ignore the 'nova/tests' path but the nova side of the definition in the > project config does try to ignore that path [8]. Interestingly a patch in > nova that only changes under the path: nova/tests/ does not trigger the job > [9]. So in this case overriding the irrelevant-files of a job works. (It > seems that overriding neutron-tempest-linuxbridge irrelevant-files works > too). > > [7] > https://github.com/openstack/neutron/blob/7e3d6a18fb928bcd303a44c1736d0d6ca9c7f0ab/.zuul.yaml#L140-L159 > [8] > https://github.com/openstack-infra/project-config/blob/5ddbd62a46e17dd2fdee07bec32aa65e3b637ff3/zuul.d/projects.yaml#L10516-L10530 > [9] https://review.openstack.org/#/c/537936/ > > I don't see what is the difference between neutron-grenade and > neutron-grenade-multinode jobs definitions from this perspective but it > seems that the irrelevent-files attribute behaves inconsistently in these > two jobs. Could you please help me undestand how irrelevant-files in > overriden jobs supposed to work? > > cheers, > gibi > > > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova][neutron][infra] zuul job definitions overrides and the irrelevant-file attribute
On Sat, Jan 27, 2018 at 2:57 AM, James E. Blair wrote: > Balázs Gibizer writes: > >> Hi, >> >> I'm getting more and more confused how the zuul job hierarchy works or >> is supposed to work. > > Hi! > > First, you (or others) may or may not have seen this already -- some of > it didn't exist when we first rolled out v3, and some of it has changed > -- but here are the relevant bits of the documentation that should help > explain what's going on. It helps to understand freezing: > > https://docs.openstack.org/infra/zuul/user/config.html#job > > and matching: > > https://docs.openstack.org/infra/zuul/user/config.html#matchers > >> First there was a bug in nova that some functional tests are not >> triggered although the job (re-)definition in the nova part of the >> project-config should not prevent it to run [1]. >> >> There we figured out that irrelevant-files parameter of the jobs are >> not something that can be overriden during re-definition or through >> parent-child relationship. The base job openstack-tox-functional has >> an irrelevant-files attribute that lists '^doc/.*$' as a path to be >> ignored [2]. In the other hand the nova part of the project-config >> tries to make this ignore less broad by adding only '^doc/source/.*$' >> . This does not work as we expected and the job did not run on changes >> that only affected ./doc/notification_samples path. We are fixing it >> by defining our own functional job in nova tree [4]. >> >> [1] https://bugs.launchpad.net/nova/+bug/1742962 >> [2] >> https://github.com/openstack-infra/openstack-zuul-jobs/blob/1823e3ea20e6dfaf37786a6ff79c56cb786bf12c/zuul.d/jobs.yaml#L380 >> [3] >> https://github.com/openstack-infra/project-config/blob/1145ab1293f5fa4d34c026856403c22b091e673c/zuul.d/projects.yaml#L10509 >> [4] https://review.openstack.org/#/c/533210/ > > This is correct. The issue here is that the irrelevant-files definition > on openstack-tox-functional is too broad. We need to be *extremely* > careful applying matchers to jobs like that. Generally I think that > irrelevant-files should be reserved for the project-pipeline invocations > only. That's how they were effectively used in Zuul v2, after all. > > Essentially, when someone puts an irrelevant-files section on a job like > that, they are saying "this job will never apply to these files, ever." > That's clearly not correct in this case. > > So our solutions are to acknowledge that it's over-broad, and reduce or > eliminate the list in [2] and expand it elsewhere (as in [3]). Or we > can say "we were generally correct, but nova is extra special so it > needs its own job". If that's the choice, then I think [4] is a fine > solution. > >> Then I started looking into other jobs to see if we made similar >> mistakes. I found two other examples in the nova related jobs where >> redefining the irrelevant-files of a job caused problems. In these >> examples nova tried to ignore more paths during the override than what >> was originally ignored in the job definition but that did not work >> [5][6]. >> >> [5] https://bugs.launchpad.net/nova/+bug/1745405 (temptest-full) > > As noted in that bug, the tempest-full job is invoked on nova via this > stanza: > > https://github.com/openstack-infra/project-config/blob/5ddbd62a46e17dd2fdee07bec32aa65e3b637ff3/zuul.d/projects.yaml#L10674-L10688 > > As expected, that did not match. There is a second invocation of > tempest-full on nova here: > > http://git.openstack.org/cgit/openstack-infra/openstack-zuul-jobs/tree/zuul.d/zuul-legacy-project-templates.yaml#n126 > > That has no irrelevant-files matches, and so matches everything. If you > drop the use of that template, it will work as expected. Or, if you can > say with some certainty that nova's irrelevant-files set is not > over-broad, you could move the irrelevant-files from nova's invocation > into the template, or even the job, and drop nova's individual > invocation. > >> [6] https://bugs.launchpad.net/nova/+bug/1745431 (neutron-grenade) > > The same template invokes this job as well. > >> So far the problem seemed to be consistent (i.e. override does not >> work). But then I looked into neutron-grenade-multinode. That job is >> defined in neutron tree (like neutron-grenade) but nova also refers to >> it in nova section of the project-config with different >> irrelevant-files than their original definition. So I assumed that >> this will lead to similar problem than in case of neutron-grenade, but >> it doesn't. >> >> The neutron-grenade-multinode original definition [1] does not try to >> ignore the 'nova/tests' path but the nova side of the definition in >> the project config does try to ignore that path [8]. Interestingly a >> patch in nova that only changes under the path: nova/tests/ does not >> trigger the job [9]. So in this case overriding the irrelevant-files >> of a job works. (It seems that overriding neutron-tempest-linuxbridge >> irrelevant-files works too). >> >> [7] >> https://github.com/o
Re: [openstack-dev] [nova][neutron][infra] zuul job definitions overrides and the irrelevant-file attribute
On Fri, Jan 26, 2018 at 6:57 PM, James E. Blair wrote: Balázs Gibizer writes: Hi, I'm getting more and more confused how the zuul job hierarchy works or is supposed to work. Hi! First, you (or others) may or may not have seen this already -- some of it didn't exist when we first rolled out v3, and some of it has changed -- but here are the relevant bits of the documentation that should help explain what's going on. It helps to understand freezing: https://docs.openstack.org/infra/zuul/user/config.html#job and matching: https://docs.openstack.org/infra/zuul/user/config.html#matchers Thanks for the doc references they are really helpful. First there was a bug in nova that some functional tests are not triggered although the job (re-)definition in the nova part of the project-config should not prevent it to run [1]. There we figured out that irrelevant-files parameter of the jobs are not something that can be overriden during re-definition or through parent-child relationship. The base job openstack-tox-functional has an irrelevant-files attribute that lists '^doc/.*$' as a path to be ignored [2]. In the other hand the nova part of the project-config tries to make this ignore less broad by adding only '^doc/source/.*$' . This does not work as we expected and the job did not run on changes that only affected ./doc/notification_samples path. We are fixing it by defining our own functional job in nova tree [4]. [1] https://bugs.launchpad.net/nova/+bug/1742962 [2] https://github.com/openstack-infra/openstack-zuul-jobs/blob/1823e3ea20e6dfaf37786a6ff79c56cb786bf12c/zuul.d/jobs.yaml#L380 [3] https://github.com/openstack-infra/project-config/blob/1145ab1293f5fa4d34c026856403c22b091e673c/zuul.d/projects.yaml#L10509 [4] https://review.openstack.org/#/c/533210/ This is correct. The issue here is that the irrelevant-files definition on openstack-tox-functional is too broad. We need to be *extremely* careful applying matchers to jobs like that. Generally I think that irrelevant-files should be reserved for the project-pipeline invocations only. That's how they were effectively used in Zuul v2, after all. Essentially, when someone puts an irrelevant-files section on a job like that, they are saying "this job will never apply to these files, ever." That's clearly not correct in this case. So our solutions are to acknowledge that it's over-broad, and reduce or eliminate the list in [2] and expand it elsewhere (as in [3]). Or we can say "we were generally correct, but nova is extra special so it needs its own job". If that's the choice, then I think [4] is a fine solution. The [4] just get merged this morning so I think that is OK for us now. Then I started looking into other jobs to see if we made similar mistakes. I found two other examples in the nova related jobs where redefining the irrelevant-files of a job caused problems. In these examples nova tried to ignore more paths during the override than what was originally ignored in the job definition but that did not work [5][6]. [5] https://bugs.launchpad.net/nova/+bug/1745405 (temptest-full) As noted in that bug, the tempest-full job is invoked on nova via this stanza: https://github.com/openstack-infra/project-config/blob/5ddbd62a46e17dd2fdee07bec32aa65e3b637ff3/zuul.d/projects.yaml#L10674-L10688 As expected, that did not match. There is a second invocation of tempest-full on nova here: http://git.openstack.org/cgit/openstack-infra/openstack-zuul-jobs/tree/zuul.d/zuul-legacy-project-templates.yaml#n126 That has no irrelevant-files matches, and so matches everything. If you drop the use of that template, it will work as expected. Or, if you can say with some certainty that nova's irrelevant-files set is not over-broad, you could move the irrelevant-files from nova's invocation into the template, or even the job, and drop nova's individual invocation. Thanks for the explanation, it is much clearer now. With this info I think I was able to propose a patcha that fixes the two bugs: https://review.openstack.org/#/c/538908/ [6] https://bugs.launchpad.net/nova/+bug/1745431 (neutron-grenade) The same template invokes this job as well. So far the problem seemed to be consistent (i.e. override does not work). But then I looked into neutron-grenade-multinode. That job is defined in neutron tree (like neutron-grenade) but nova also refers to it in nova section of the project-config with different irrelevant-files than their original definition. So I assumed that this will lead to similar problem than in case of neutron-grenade, but it doesn't. The neutron-grenade-multinode original definition [1] does not try to ignore the 'nova/tests' path but the nova side of the definition in the project config does try to ignore that path [8]. Interestingly a patch in nova that only changes under the path: nova/tests/ does not trigger the job [9]. So in th
Re: [openstack-dev] [nova][neutron][infra] zuul job definitions overrides and the irrelevant-file attribute
Balázs Gibizer writes: > Hi, > > I'm getting more and more confused how the zuul job hierarchy works or > is supposed to work. Hi! First, you (or others) may or may not have seen this already -- some of it didn't exist when we first rolled out v3, and some of it has changed -- but here are the relevant bits of the documentation that should help explain what's going on. It helps to understand freezing: https://docs.openstack.org/infra/zuul/user/config.html#job and matching: https://docs.openstack.org/infra/zuul/user/config.html#matchers > First there was a bug in nova that some functional tests are not > triggered although the job (re-)definition in the nova part of the > project-config should not prevent it to run [1]. > > There we figured out that irrelevant-files parameter of the jobs are > not something that can be overriden during re-definition or through > parent-child relationship. The base job openstack-tox-functional has > an irrelevant-files attribute that lists '^doc/.*$' as a path to be > ignored [2]. In the other hand the nova part of the project-config > tries to make this ignore less broad by adding only '^doc/source/.*$' > . This does not work as we expected and the job did not run on changes > that only affected ./doc/notification_samples path. We are fixing it > by defining our own functional job in nova tree [4]. > > [1] https://bugs.launchpad.net/nova/+bug/1742962 > [2] > https://github.com/openstack-infra/openstack-zuul-jobs/blob/1823e3ea20e6dfaf37786a6ff79c56cb786bf12c/zuul.d/jobs.yaml#L380 > [3] > https://github.com/openstack-infra/project-config/blob/1145ab1293f5fa4d34c026856403c22b091e673c/zuul.d/projects.yaml#L10509 > [4] https://review.openstack.org/#/c/533210/ This is correct. The issue here is that the irrelevant-files definition on openstack-tox-functional is too broad. We need to be *extremely* careful applying matchers to jobs like that. Generally I think that irrelevant-files should be reserved for the project-pipeline invocations only. That's how they were effectively used in Zuul v2, after all. Essentially, when someone puts an irrelevant-files section on a job like that, they are saying "this job will never apply to these files, ever." That's clearly not correct in this case. So our solutions are to acknowledge that it's over-broad, and reduce or eliminate the list in [2] and expand it elsewhere (as in [3]). Or we can say "we were generally correct, but nova is extra special so it needs its own job". If that's the choice, then I think [4] is a fine solution. > Then I started looking into other jobs to see if we made similar > mistakes. I found two other examples in the nova related jobs where > redefining the irrelevant-files of a job caused problems. In these > examples nova tried to ignore more paths during the override than what > was originally ignored in the job definition but that did not work > [5][6]. > > [5] https://bugs.launchpad.net/nova/+bug/1745405 (temptest-full) As noted in that bug, the tempest-full job is invoked on nova via this stanza: https://github.com/openstack-infra/project-config/blob/5ddbd62a46e17dd2fdee07bec32aa65e3b637ff3/zuul.d/projects.yaml#L10674-L10688 As expected, that did not match. There is a second invocation of tempest-full on nova here: http://git.openstack.org/cgit/openstack-infra/openstack-zuul-jobs/tree/zuul.d/zuul-legacy-project-templates.yaml#n126 That has no irrelevant-files matches, and so matches everything. If you drop the use of that template, it will work as expected. Or, if you can say with some certainty that nova's irrelevant-files set is not over-broad, you could move the irrelevant-files from nova's invocation into the template, or even the job, and drop nova's individual invocation. > [6] https://bugs.launchpad.net/nova/+bug/1745431 (neutron-grenade) The same template invokes this job as well. > So far the problem seemed to be consistent (i.e. override does not > work). But then I looked into neutron-grenade-multinode. That job is > defined in neutron tree (like neutron-grenade) but nova also refers to > it in nova section of the project-config with different > irrelevant-files than their original definition. So I assumed that > this will lead to similar problem than in case of neutron-grenade, but > it doesn't. > > The neutron-grenade-multinode original definition [1] does not try to > ignore the 'nova/tests' path but the nova side of the definition in > the project config does try to ignore that path [8]. Interestingly a > patch in nova that only changes under the path: nova/tests/ does not > trigger the job [9]. So in this case overriding the irrelevant-files > of a job works. (It seems that overriding neutron-tempest-linuxbridge > irrelevant-files works too). > > [7] > https://github.com/openstack/neutron/blob/7e3d6a18fb928bcd303a44c1736d0d6ca9c7f0ab/.zuul.yaml#L140-L159 > [8] > https://github.com/openstack-infra/project-config/blob/5ddbd62a46e17dd2fdee07bec32aa65e3b637ff3/zuul.d/p
[openstack-dev] [nova][neutron][infra] zuul job definitions overrides and the irrelevant-file attribute
Hi, I'm getting more and more confused how the zuul job hierarchy works or is supposed to work. First there was a bug in nova that some functional tests are not triggered although the job (re-)definition in the nova part of the project-config should not prevent it to run [1]. There we figured out that irrelevant-files parameter of the jobs are not something that can be overriden during re-definition or through parent-child relationship. The base job openstack-tox-functional has an irrelevant-files attribute that lists '^doc/.*$' as a path to be ignored [2]. In the other hand the nova part of the project-config tries to make this ignore less broad by adding only '^doc/source/.*$' . This does not work as we expected and the job did not run on changes that only affected ./doc/notification_samples path. We are fixing it by defining our own functional job in nova tree [4]. [1] https://bugs.launchpad.net/nova/+bug/1742962 [2] https://github.com/openstack-infra/openstack-zuul-jobs/blob/1823e3ea20e6dfaf37786a6ff79c56cb786bf12c/zuul.d/jobs.yaml#L380 [3] https://github.com/openstack-infra/project-config/blob/1145ab1293f5fa4d34c026856403c22b091e673c/zuul.d/projects.yaml#L10509 [4] https://review.openstack.org/#/c/533210/ Then I started looking into other jobs to see if we made similar mistakes. I found two other examples in the nova related jobs where redefining the irrelevant-files of a job caused problems. In these examples nova tried to ignore more paths during the override than what was originally ignored in the job definition but that did not work [5][6]. [5] https://bugs.launchpad.net/nova/+bug/1745405 (temptest-full) [6] https://bugs.launchpad.net/nova/+bug/1745431 (neutron-grenade) So far the problem seemed to be consistent (i.e. override does not work). But then I looked into neutron-grenade-multinode. That job is defined in neutron tree (like neutron-grenade) but nova also refers to it in nova section of the project-config with different irrelevant-files than their original definition. So I assumed that this will lead to similar problem than in case of neutron-grenade, but it doesn't. The neutron-grenade-multinode original definition [1] does not try to ignore the 'nova/tests' path but the nova side of the definition in the project config does try to ignore that path [8]. Interestingly a patch in nova that only changes under the path: nova/tests/ does not trigger the job [9]. So in this case overriding the irrelevant-files of a job works. (It seems that overriding neutron-tempest-linuxbridge irrelevant-files works too). [7] https://github.com/openstack/neutron/blob/7e3d6a18fb928bcd303a44c1736d0d6ca9c7f0ab/.zuul.yaml#L140-L159 [8] https://github.com/openstack-infra/project-config/blob/5ddbd62a46e17dd2fdee07bec32aa65e3b637ff3/zuul.d/projects.yaml#L10516-L10530 [9] https://review.openstack.org/#/c/537936/ I don't see what is the difference between neutron-grenade and neutron-grenade-multinode jobs definitions from this perspective but it seems that the irrelevent-files attribute behaves inconsistently in these two jobs. Could you please help me undestand how irrelevant-files in overriden jobs supposed to work? cheers, gibi __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
