Re: [openstack-dev] [nova][qa] Do we turn on voting for the tempest-dsvm-cells job?
On 06/23/2015 08:13 AM, John Garbutt wrote: The question for the nova team is, shall we make the tempest-dsvm-cells job voting on nova changes knowing that the gate can be broken with a change to tempest that isn't caught in the regex? In my opinion I think we should make it voting so we don't regress cells with changes to nova that go unnoticed with the non-voting job today. Cells v2 is a nova priority for Liberty so we don't want setbacks now that it's stable. I would be tempted to risk it, given the large gain, but I am a little biased too. But with us controlling the regex, that seems a much easier call to say yes. Right, with the REGEX back in control of the Nova team, we can just temporarily drop any failing tests from Tempest changes until fixes or reverts happen. I honestly think this kind of asymetric decoupling is a thing we're going to need to get good at, because everything can't cross gate with everything else. Otherwise we'll go back to 100 hr gate queues and bugs that take a week to resolve in the intertwining. -Sean -- Sean Dague http://dague.net __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova][qa] Do we turn on voting for the tempest-dsvm-cells job?
On 22 June 2015 at 23:03, Matt Riedemann mrie...@linux.vnet.ibm.com wrote: On 6/22/2015 4:38 PM, Matt Riedemann wrote: On 6/22/2015 4:32 PM, Russell Bryant wrote: On 06/22/2015 05:23 PM, Matt Riedemann wrote: The check-tempest-dsvm-cells job has been in nova's check queue since January as non-voting and has been stable for a couple of weeks now, so before it's regressed melwitt proposed a change to making it voting and gating on nova changes [1]. I raised a concern in that change that the tempest-dsvm-cells job is not in the check queue for tempest or devstack changes, so if a change is merged in tempest/devstack which breaks the cells job, it will block nova changes from merging. mtreinish noted that tempest already has around 30 jobs running against it right now in the check queue, so he'd prefer that another one isn't added since the nova API shouldn't be different in the case of cells, but we know there are quirks. That can be seen from the massive regex of excluded tests for the tempest-dvsm-cells job [2]. That sounds like a good call. For the record, cells v2 should fix this major issue, which is awesome. If we could turn that regex list into tempest configurations, I think that would make it possible to not have to run tempest changes through the cells job in the check queue but also feel reasonably confident that changes to tempest that use the config options properly won't break the cells job (and block nova in the gate). This is actually something we should do regardless of voting or not on nova since new tests get added which might not fall in the regex and break the cells job. So I'm going to propose some changes so that the regex will be moved to devstack-gate (regex exodus (tm)) and we'll work on whittling down the regex there (and run those d-g changes against the tempest-dsvm-cells job in the experimental queue). The question for the nova team is, shall we make the tempest-dsvm-cells job voting on nova changes knowing that the gate can be broken with a change to tempest that isn't caught in the regex? In my opinion I think we should make it voting so we don't regress cells with changes to nova that go unnoticed with the non-voting job today. Cells v2 is a nova priority for Liberty so we don't want setbacks now that it's stable. I would be tempted to risk it, given the large gain, but I am a little biased too. But with us controlling the regex, that seems a much easier call to say yes. If a change does land in tempest which breaks the cells job and blocks nova, we (1) fix it or (2) modify the regex so it's excluded until fixed as has been done up to this point. I think we should probably mull this over in the ML and then vote on it in this week's nova meeting. [1] https://review.openstack.org/#/c/190894/ [2] http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/jobs/devstack-gate.yaml#n1004 Regarding your regex exodus, I recently added something for this. In another project, I'm setting the regex in a file I keep in the code repo instead of project-config. support for DEVSTACK_GATE_SETTINGS in devstack-gate: https://review.openstack.org/190321 usage in a job definition: https://review.openstack.org/190325 a DEVSTACK_GATE_SETTINGS file that sets DEVSTACK_GATE_TEMPEST_REGEX: https://review.openstack.org/186894 It all seems to be working for me, except I still need to tweak my regex to get the job passing, but at least I can do that without updating project-config now. Awesome, that is way cleaner. I'll go that route instead, thanks! Here is the change that moves the regex into nova: https://review.openstack.org/#/c/194411/ This seems like a good best of both worlds. Awesome stuff. Thanks, John __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova][qa] Do we turn on voting for the tempest-dsvm-cells job?
On 6/22/2015 4:32 PM, Russell Bryant wrote: On 06/22/2015 05:23 PM, Matt Riedemann wrote: The check-tempest-dsvm-cells job has been in nova's check queue since January as non-voting and has been stable for a couple of weeks now, so before it's regressed melwitt proposed a change to making it voting and gating on nova changes [1]. I raised a concern in that change that the tempest-dsvm-cells job is not in the check queue for tempest or devstack changes, so if a change is merged in tempest/devstack which breaks the cells job, it will block nova changes from merging. mtreinish noted that tempest already has around 30 jobs running against it right now in the check queue, so he'd prefer that another one isn't added since the nova API shouldn't be different in the case of cells, but we know there are quirks. That can be seen from the massive regex of excluded tests for the tempest-dvsm-cells job [2]. If we could turn that regex list into tempest configurations, I think that would make it possible to not have to run tempest changes through the cells job in the check queue but also feel reasonably confident that changes to tempest that use the config options properly won't break the cells job (and block nova in the gate). This is actually something we should do regardless of voting or not on nova since new tests get added which might not fall in the regex and break the cells job. So I'm going to propose some changes so that the regex will be moved to devstack-gate (regex exodus (tm)) and we'll work on whittling down the regex there (and run those d-g changes against the tempest-dsvm-cells job in the experimental queue). The question for the nova team is, shall we make the tempest-dsvm-cells job voting on nova changes knowing that the gate can be broken with a change to tempest that isn't caught in the regex? In my opinion I think we should make it voting so we don't regress cells with changes to nova that go unnoticed with the non-voting job today. Cells v2 is a nova priority for Liberty so we don't want setbacks now that it's stable. If a change does land in tempest which breaks the cells job and blocks nova, we (1) fix it or (2) modify the regex so it's excluded until fixed as has been done up to this point. I think we should probably mull this over in the ML and then vote on it in this week's nova meeting. [1] https://review.openstack.org/#/c/190894/ [2] http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/jobs/devstack-gate.yaml#n1004 Regarding your regex exodus, I recently added something for this. In another project, I'm setting the regex in a file I keep in the code repo instead of project-config. support for DEVSTACK_GATE_SETTINGS in devstack-gate: https://review.openstack.org/190321 usage in a job definition: https://review.openstack.org/190325 a DEVSTACK_GATE_SETTINGS file that sets DEVSTACK_GATE_TEMPEST_REGEX: https://review.openstack.org/186894 It all seems to be working for me, except I still need to tweak my regex to get the job passing, but at least I can do that without updating project-config now. Awesome, that is way cleaner. I'll go that route instead, thanks! -- Thanks, Matt Riedemann __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [nova][qa] Do we turn on voting for the tempest-dsvm-cells job?
The check-tempest-dsvm-cells job has been in nova's check queue since January as non-voting and has been stable for a couple of weeks now, so before it's regressed melwitt proposed a change to making it voting and gating on nova changes [1]. I raised a concern in that change that the tempest-dsvm-cells job is not in the check queue for tempest or devstack changes, so if a change is merged in tempest/devstack which breaks the cells job, it will block nova changes from merging. mtreinish noted that tempest already has around 30 jobs running against it right now in the check queue, so he'd prefer that another one isn't added since the nova API shouldn't be different in the case of cells, but we know there are quirks. That can be seen from the massive regex of excluded tests for the tempest-dvsm-cells job [2]. If we could turn that regex list into tempest configurations, I think that would make it possible to not have to run tempest changes through the cells job in the check queue but also feel reasonably confident that changes to tempest that use the config options properly won't break the cells job (and block nova in the gate). This is actually something we should do regardless of voting or not on nova since new tests get added which might not fall in the regex and break the cells job. So I'm going to propose some changes so that the regex will be moved to devstack-gate (regex exodus (tm)) and we'll work on whittling down the regex there (and run those d-g changes against the tempest-dsvm-cells job in the experimental queue). The question for the nova team is, shall we make the tempest-dsvm-cells job voting on nova changes knowing that the gate can be broken with a change to tempest that isn't caught in the regex? In my opinion I think we should make it voting so we don't regress cells with changes to nova that go unnoticed with the non-voting job today. Cells v2 is a nova priority for Liberty so we don't want setbacks now that it's stable. If a change does land in tempest which breaks the cells job and blocks nova, we (1) fix it or (2) modify the regex so it's excluded until fixed as has been done up to this point. I think we should probably mull this over in the ML and then vote on it in this week's nova meeting. [1] https://review.openstack.org/#/c/190894/ [2] http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/jobs/devstack-gate.yaml#n1004 -- Thanks, Matt Riedemann __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova][qa] Do we turn on voting for the tempest-dsvm-cells job?
On 06/22/2015 05:23 PM, Matt Riedemann wrote: The check-tempest-dsvm-cells job has been in nova's check queue since January as non-voting and has been stable for a couple of weeks now, so before it's regressed melwitt proposed a change to making it voting and gating on nova changes [1]. I raised a concern in that change that the tempest-dsvm-cells job is not in the check queue for tempest or devstack changes, so if a change is merged in tempest/devstack which breaks the cells job, it will block nova changes from merging. mtreinish noted that tempest already has around 30 jobs running against it right now in the check queue, so he'd prefer that another one isn't added since the nova API shouldn't be different in the case of cells, but we know there are quirks. That can be seen from the massive regex of excluded tests for the tempest-dvsm-cells job [2]. If we could turn that regex list into tempest configurations, I think that would make it possible to not have to run tempest changes through the cells job in the check queue but also feel reasonably confident that changes to tempest that use the config options properly won't break the cells job (and block nova in the gate). This is actually something we should do regardless of voting or not on nova since new tests get added which might not fall in the regex and break the cells job. So I'm going to propose some changes so that the regex will be moved to devstack-gate (regex exodus (tm)) and we'll work on whittling down the regex there (and run those d-g changes against the tempest-dsvm-cells job in the experimental queue). The question for the nova team is, shall we make the tempest-dsvm-cells job voting on nova changes knowing that the gate can be broken with a change to tempest that isn't caught in the regex? In my opinion I think we should make it voting so we don't regress cells with changes to nova that go unnoticed with the non-voting job today. Cells v2 is a nova priority for Liberty so we don't want setbacks now that it's stable. If a change does land in tempest which breaks the cells job and blocks nova, we (1) fix it or (2) modify the regex so it's excluded until fixed as has been done up to this point. I think we should probably mull this over in the ML and then vote on it in this week's nova meeting. [1] https://review.openstack.org/#/c/190894/ [2] http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/jobs/devstack-gate.yaml#n1004 Regarding your regex exodus, I recently added something for this. In another project, I'm setting the regex in a file I keep in the code repo instead of project-config. support for DEVSTACK_GATE_SETTINGS in devstack-gate: https://review.openstack.org/190321 usage in a job definition: https://review.openstack.org/190325 a DEVSTACK_GATE_SETTINGS file that sets DEVSTACK_GATE_TEMPEST_REGEX: https://review.openstack.org/186894 It all seems to be working for me, except I still need to tweak my regex to get the job passing, but at least I can do that without updating project-config now. -- Russell Bryant __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova][qa] Do we turn on voting for the tempest-dsvm-cells job?
On 06/22/15 at 04:23pm, Matt Riedemann wrote: The check-tempest-dsvm-cells job has been in nova's check queue since January as non-voting and has been stable for a couple of weeks now, so before it's regressed melwitt proposed a change to making it voting and gating on nova changes [1]. I raised a concern in that change that the tempest-dsvm-cells job is not in the check queue for tempest or devstack changes, so if a change is merged in tempest/devstack which breaks the cells job, it will block nova changes from merging. mtreinish noted that tempest already has around 30 jobs running against it right now in the check queue, so he'd prefer that another one isn't added since the nova API shouldn't be different in the case of cells, but we know there are quirks. That can be seen from the massive regex of excluded tests for the tempest-dvsm-cells job [2]. If we could turn that regex list into tempest configurations, I think that would make it possible to not have to run tempest changes through the cells job in the check queue but also feel reasonably confident that changes to tempest that use the config options properly won't break the cells job (and block nova in the gate). This is actually something we should do regardless of voting or not on nova since new tests get added which might not fall in the regex and break the cells job. So I'm going to propose some changes so that the regex will be moved to devstack-gate (regex exodus (tm)) and we'll work on whittling down the regex there (and run those d-g changes against the tempest-dsvm-cells job in the experimental queue). The question for the nova team is, shall we make the tempest-dsvm-cells job voting on nova changes knowing that the gate can be broken with a change to tempest that isn't caught in the regex? In my opinion I think we should make it voting so we don't regress cells with changes to nova that go unnoticed with the non-voting job today. Cells v2 is a nova priority for Liberty so we don't want setbacks now that it's stable. I agree, though I'm pretty biased. If a change does land in tempest which breaks the cells job and blocks nova, we (1) fix it or (2) modify the regex so it's excluded until fixed as has been done up to this point. During our efforts last cycle to reduce the number of failing tests it seemed that we had regressions based on Tempest changes 2 times. Once around security group logic being added to tests and again when the networking setup changed during tests that created an instance. The security group change led to exclusions and the networking change we were able to fix with a devstack change, and a Tempest flag iirc. I think we should probably mull this over in the ML and then vote on it in this week's nova meeting. [1] https://review.openstack.org/#/c/190894/ [2] http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/jobs/devstack-gate.yaml#n1004 -- Thanks, Matt Riedemann __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova][qa] Do we turn on voting for the tempest-dsvm-cells job?
On 6/22/2015 4:38 PM, Matt Riedemann wrote: On 6/22/2015 4:32 PM, Russell Bryant wrote: On 06/22/2015 05:23 PM, Matt Riedemann wrote: The check-tempest-dsvm-cells job has been in nova's check queue since January as non-voting and has been stable for a couple of weeks now, so before it's regressed melwitt proposed a change to making it voting and gating on nova changes [1]. I raised a concern in that change that the tempest-dsvm-cells job is not in the check queue for tempest or devstack changes, so if a change is merged in tempest/devstack which breaks the cells job, it will block nova changes from merging. mtreinish noted that tempest already has around 30 jobs running against it right now in the check queue, so he'd prefer that another one isn't added since the nova API shouldn't be different in the case of cells, but we know there are quirks. That can be seen from the massive regex of excluded tests for the tempest-dvsm-cells job [2]. If we could turn that regex list into tempest configurations, I think that would make it possible to not have to run tempest changes through the cells job in the check queue but also feel reasonably confident that changes to tempest that use the config options properly won't break the cells job (and block nova in the gate). This is actually something we should do regardless of voting or not on nova since new tests get added which might not fall in the regex and break the cells job. So I'm going to propose some changes so that the regex will be moved to devstack-gate (regex exodus (tm)) and we'll work on whittling down the regex there (and run those d-g changes against the tempest-dsvm-cells job in the experimental queue). The question for the nova team is, shall we make the tempest-dsvm-cells job voting on nova changes knowing that the gate can be broken with a change to tempest that isn't caught in the regex? In my opinion I think we should make it voting so we don't regress cells with changes to nova that go unnoticed with the non-voting job today. Cells v2 is a nova priority for Liberty so we don't want setbacks now that it's stable. If a change does land in tempest which breaks the cells job and blocks nova, we (1) fix it or (2) modify the regex so it's excluded until fixed as has been done up to this point. I think we should probably mull this over in the ML and then vote on it in this week's nova meeting. [1] https://review.openstack.org/#/c/190894/ [2] http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/jobs/devstack-gate.yaml#n1004 Regarding your regex exodus, I recently added something for this. In another project, I'm setting the regex in a file I keep in the code repo instead of project-config. support for DEVSTACK_GATE_SETTINGS in devstack-gate: https://review.openstack.org/190321 usage in a job definition: https://review.openstack.org/190325 a DEVSTACK_GATE_SETTINGS file that sets DEVSTACK_GATE_TEMPEST_REGEX: https://review.openstack.org/186894 It all seems to be working for me, except I still need to tweak my regex to get the job passing, but at least I can do that without updating project-config now. Awesome, that is way cleaner. I'll go that route instead, thanks! Here is the change that moves the regex into nova: https://review.openstack.org/#/c/194411/ -- Thanks, Matt Riedemann __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
