Re: [openstack-dev] [swift] [qa] which test configs does the swift team find useful
On 11/25/2014 02:46 PM, John Dickinson wrote: > This is great! > > Sean, I agree with your analysis. > > gate-swift-pep8 (yes) > gate-swift-docs (yes) > gate-swift-python27 (yes) > gate-swift-tox-func (yes) > check-swift-dsvm-functional (yes) > check-tempest-dsvm-full(to further ensure glance/heat/cinder checking) > check-grenade-dsvm (I can go either way on this one, I won't fight for > or against it) Ok, great - https://review.openstack.org/#/c/137184/ should implement that analysis if I got the patch right. -Sean -- Sean Dague http://dague.net signature.asc Description: OpenPGP digital signature ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [swift] [qa] which test configs does the swift team find useful
This is great! Sean, I agree with your analysis. gate-swift-pep8 (yes) gate-swift-docs (yes) gate-swift-python27 (yes) gate-swift-tox-func (yes) check-swift-dsvm-functional (yes) check-tempest-dsvm-full(to further ensure glance/heat/cinder checking) check-grenade-dsvm (I can go either way on this one, I won't fight for or against it) --John > On Nov 25, 2014, at 7:03 AM, Sean Dague wrote: > > As we are trying to do smart disaggregation of tests in the gate, I > think it's important to figure out which test configurations seem to be > actually helping, and which aren't. As the swift team has long had a > functional test job, this seems like a good place to start. (Also the > field deploy / upgrade story on Swift is probably one of the best of any > OpenStack project, so removing friction is probably in order.) > > gate-swift-pep8 SUCCESS in 1m 16s > gate-swift-docs SUCCESS in 1m 48s > gate-swift-python27 SUCCESS in 3m 24s > check-tempest-dsvm-full SUCCESS in 56m 51s > check-tempest-dsvm-postgres-full SUCCESS in 54m 53s > check-tempest-dsvm-neutron-full SUCCESS in 1h 06m 09s > check-tempest-dsvm-neutron-heat-slow SUCCESS in 31m 18s > check-grenade-dsvmSUCCESS in 39m 33s > gate-tempest-dsvm-large-ops SUCCESS in 29m 34s > gate-tempest-dsvm-neutron-large-ops SUCCESS in 22m 11s > gate-swift-tox-func SUCCESS in 2m 50s (non-voting) > check-swift-dsvm-functional SUCCESS in 17m 12s > check-devstack-dsvm-cells SUCCESS in 15m 18s > > > I think in looking at that it's obvious that: > * check-devstack-dsvm-cells > * check-tempest-dsvm-postgres-full > * gate-tempest-dsvm-large-ops > * gate-tempest-dsvm-neutron-large-ops > * check-tempest-dsvm-neutron-full > > Provide nothing new to swift, the access patterns on the glance => swift > interaction aren't impacted on any of those, neither is the heat / swift > resource tests or volumes / swift backup tests. > > check-tempest-dsvm-neutron-heat-slow doesn't touch swift either (it's > actually remarkably sparse of any content). > > Which kind of leaves us with 1 full stack run, and the grenade job. Have > those caught real bugs? Does there remain value in them? Have other > teams that rely on swift found those to block regressions? > > Let's figure out what's helpful, and what's not, and purge out all the > non helpful stuff. > > -Sean > > -- > Sean Dague > http://dague.net > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev signature.asc Description: Message signed with OpenPGP using GPGMail ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [swift] [qa] which test configs does the swift team find useful
On Tue, Nov 25, 2014 at 11:01:04AM -0500, Sean Dague wrote: > On 11/25/2014 10:54 AM, Matthew Treinish wrote: > > On Tue, Nov 25, 2014 at 10:03:41AM -0500, Sean Dague wrote: > >> As we are trying to do smart disaggregation of tests in the gate, I > >> think it's important to figure out which test configurations seem to be > >> actually helping, and which aren't. As the swift team has long had a > >> functional test job, this seems like a good place to start. (Also the > >> field deploy / upgrade story on Swift is probably one of the best of any > >> OpenStack project, so removing friction is probably in order.) > >> > >> gate-swift-pep8SUCCESS in 1m 16s > >> gate-swift-docsSUCCESS in 1m 48s > >> gate-swift-python27SUCCESS in 3m 24s > >> check-tempest-dsvm-fullSUCCESS in 56m 51s > >> check-tempest-dsvm-postgres-full SUCCESS in 54m 53s > >> check-tempest-dsvm-neutron-fullSUCCESS in 1h 06m 09s > >> check-tempest-dsvm-neutron-heat-slow SUCCESS in 31m 18s > >> check-grenade-dsvm SUCCESS in 39m 33s > >> gate-tempest-dsvm-large-opsSUCCESS in 29m 34s > >> gate-tempest-dsvm-neutron-large-opsSUCCESS in 22m 11s > >> gate-swift-tox-funcSUCCESS in 2m 50s (non-voting) > >> check-swift-dsvm-functionalSUCCESS in 17m 12s > >> check-devstack-dsvm-cells SUCCESS in 15m 18s > >> > >> > >> I think in looking at that it's obvious that: > >> * check-devstack-dsvm-cells > >> * check-tempest-dsvm-postgres-full > >> * gate-tempest-dsvm-large-ops > >> * gate-tempest-dsvm-neutron-large-ops > >> * check-tempest-dsvm-neutron-full > >> > >> Provide nothing new to swift, the access patterns on the glance => swift > >> interaction aren't impacted on any of those, neither is the heat / swift > >> resource tests or volumes / swift backup tests. > > > > So I agree with all of this and think removing the jobs is fine, except the > > postgres job and the neutron jobs do test the glance->swift access pattern, > > and > > do run the heat swift tests. But, it does raise the bigger question which > > was > > brought up in Darmstadt and again at summit on having a single gating > > configuration. Maybe we should just switch to doing that now and finally > > drop > > the postgres job completely. > > I'm not saying it doesn't test it. I'm saying it doesn't test it in any > way which is different from the mysql job. > > "Provide nothing new to swift"... Sure, but I think it raises the bigger question around the postgres jobs in general, because I think if we're having this conversation around swift it really applies to all of the postgres jobs. I think it's probably time that we just move postgres jobs to periodic/experimental everywhere and be done with it. > > >> check-tempest-dsvm-neutron-heat-slow doesn't touch swift either (it's > >> actually remarkably sparse of any content). > > > > I think we probably should be removing this job from everywhere, we've > > slowly > > been whittling away the job because it doesn't seem to be capable of being > > run > > reliably. This also doesn't run any swift resource tests, in it's current > > form > > it runs 6 neutron resource tests and thats it. > > There is a separate thread on that as well. > > >> Which kind of leaves us with 1 full stack run, and the grenade job. Have > >> those caught real bugs? Does there remain value in them? Have other > >> teams that rely on swift found those to block regressions? > > > > So I think we'll need these at a minimum for the time being. Giving our > > current > > project structure (and governance requirements) having jobs that test things > > work together I feel is important. I know we've caught issues with > > glance->swift > > layer with these jobs in the past as well as other bugs as well as bugs in > > swift > > before. (although they're very infrequent compared to other projects) > > > >> > >> Let's figure out what's helpful, and what's not, and purge out all the > >> non helpful stuff. > >> -Matt Treinish pgpJm1miWqaKr.pgp Description: PGP signature ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [swift] [qa] which test configs does the swift team find useful
On 11/25/2014 10:54 AM, Matthew Treinish wrote: > On Tue, Nov 25, 2014 at 10:03:41AM -0500, Sean Dague wrote: >> As we are trying to do smart disaggregation of tests in the gate, I >> think it's important to figure out which test configurations seem to be >> actually helping, and which aren't. As the swift team has long had a >> functional test job, this seems like a good place to start. (Also the >> field deploy / upgrade story on Swift is probably one of the best of any >> OpenStack project, so removing friction is probably in order.) >> >> gate-swift-pep8 SUCCESS in 1m 16s >> gate-swift-docs SUCCESS in 1m 48s >> gate-swift-python27 SUCCESS in 3m 24s >> check-tempest-dsvm-full SUCCESS in 56m 51s >> check-tempest-dsvm-postgres-full SUCCESS in 54m 53s >> check-tempest-dsvm-neutron-full SUCCESS in 1h 06m 09s >> check-tempest-dsvm-neutron-heat-slow SUCCESS in 31m 18s >> check-grenade-dsvm SUCCESS in 39m 33s >> gate-tempest-dsvm-large-ops SUCCESS in 29m 34s >> gate-tempest-dsvm-neutron-large-ops SUCCESS in 22m 11s >> gate-swift-tox-func SUCCESS in 2m 50s (non-voting) >> check-swift-dsvm-functional SUCCESS in 17m 12s >> check-devstack-dsvm-cellsSUCCESS in 15m 18s >> >> >> I think in looking at that it's obvious that: >> * check-devstack-dsvm-cells >> * check-tempest-dsvm-postgres-full >> * gate-tempest-dsvm-large-ops >> * gate-tempest-dsvm-neutron-large-ops >> * check-tempest-dsvm-neutron-full >> >> Provide nothing new to swift, the access patterns on the glance => swift >> interaction aren't impacted on any of those, neither is the heat / swift >> resource tests or volumes / swift backup tests. > > So I agree with all of this and think removing the jobs is fine, except the > postgres job and the neutron jobs do test the glance->swift access pattern, > and > do run the heat swift tests. But, it does raise the bigger question which was > brought up in Darmstadt and again at summit on having a single gating > configuration. Maybe we should just switch to doing that now and finally drop > the postgres job completely. I'm not saying it doesn't test it. I'm saying it doesn't test it in any way which is different from the mysql job. "Provide nothing new to swift"... >> check-tempest-dsvm-neutron-heat-slow doesn't touch swift either (it's >> actually remarkably sparse of any content). > > I think we probably should be removing this job from everywhere, we've slowly > been whittling away the job because it doesn't seem to be capable of being run > reliably. This also doesn't run any swift resource tests, in it's current form > it runs 6 neutron resource tests and thats it. There is a separate thread on that as well. >> Which kind of leaves us with 1 full stack run, and the grenade job. Have >> those caught real bugs? Does there remain value in them? Have other >> teams that rely on swift found those to block regressions? > > So I think we'll need these at a minimum for the time being. Giving our > current > project structure (and governance requirements) having jobs that test things > work together I feel is important. I know we've caught issues with > glance->swift > layer with these jobs in the past as well as other bugs as well as bugs in > swift > before. (although they're very infrequent compared to other projects) > >> >> Let's figure out what's helpful, and what's not, and purge out all the >> non helpful stuff. >> > > -Matt Treinish > > > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > -- Sean Dague http://dague.net signature.asc Description: OpenPGP digital signature ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [swift] [qa] which test configs does the swift team find useful
On Tue, Nov 25, 2014 at 10:03:41AM -0500, Sean Dague wrote: > As we are trying to do smart disaggregation of tests in the gate, I > think it's important to figure out which test configurations seem to be > actually helping, and which aren't. As the swift team has long had a > functional test job, this seems like a good place to start. (Also the > field deploy / upgrade story on Swift is probably one of the best of any > OpenStack project, so removing friction is probably in order.) > > gate-swift-pep8 SUCCESS in 1m 16s > gate-swift-docs SUCCESS in 1m 48s > gate-swift-python27 SUCCESS in 3m 24s > check-tempest-dsvm-full SUCCESS in 56m 51s > check-tempest-dsvm-postgres-full SUCCESS in 54m 53s > check-tempest-dsvm-neutron-full SUCCESS in 1h 06m 09s > check-tempest-dsvm-neutron-heat-slow SUCCESS in 31m 18s > check-grenade-dsvmSUCCESS in 39m 33s > gate-tempest-dsvm-large-ops SUCCESS in 29m 34s > gate-tempest-dsvm-neutron-large-ops SUCCESS in 22m 11s > gate-swift-tox-func SUCCESS in 2m 50s (non-voting) > check-swift-dsvm-functional SUCCESS in 17m 12s > check-devstack-dsvm-cells SUCCESS in 15m 18s > > > I think in looking at that it's obvious that: > * check-devstack-dsvm-cells > * check-tempest-dsvm-postgres-full > * gate-tempest-dsvm-large-ops > * gate-tempest-dsvm-neutron-large-ops > * check-tempest-dsvm-neutron-full > > Provide nothing new to swift, the access patterns on the glance => swift > interaction aren't impacted on any of those, neither is the heat / swift > resource tests or volumes / swift backup tests. So I agree with all of this and think removing the jobs is fine, except the postgres job and the neutron jobs do test the glance->swift access pattern, and do run the heat swift tests. But, it does raise the bigger question which was brought up in Darmstadt and again at summit on having a single gating configuration. Maybe we should just switch to doing that now and finally drop the postgres job completely. > > check-tempest-dsvm-neutron-heat-slow doesn't touch swift either (it's > actually remarkably sparse of any content). I think we probably should be removing this job from everywhere, we've slowly been whittling away the job because it doesn't seem to be capable of being run reliably. This also doesn't run any swift resource tests, in it's current form it runs 6 neutron resource tests and thats it. > > Which kind of leaves us with 1 full stack run, and the grenade job. Have > those caught real bugs? Does there remain value in them? Have other > teams that rely on swift found those to block regressions? So I think we'll need these at a minimum for the time being. Giving our current project structure (and governance requirements) having jobs that test things work together I feel is important. I know we've caught issues with glance->swift layer with these jobs in the past as well as other bugs as well as bugs in swift before. (although they're very infrequent compared to other projects) > > Let's figure out what's helpful, and what's not, and purge out all the > non helpful stuff. > -Matt Treinish pgpRu_dsPCbCn.pgp Description: PGP signature ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
