Re: [openstack-dev] [devstack] plugin: Giving ability for plugin repo to override tempest tests

[Andrey M . Pavlov]

Deepak, Our gating config is here -https://github.com/openstack-infra/project-config/blob/master/jenkins/jobs/ec2-api.yamlPlease see first template.This is just a copy of general tempest job with our settings. Andrey 15.04.2015, 13:56, "Deepak Shetty" :On Wed, Apr 15, 2015 at 3:59 PM, Sean Dague  wrote:On 04/15/2015 01:51 AM, Andrey M. Pavlov wrote: > Hi, > > We have similar situation with our stackforge/ec2-api project. > Patches in openstack repos can break our plugin. > But I don't see any possibility until our project not in OpenStack repo. > And I don't understand how plugins for tempest can help to us. > > My thoughts is only ec2-api gating jobs(this is implemented now in our > gating): > - make own tempest job with plugin definition and restrict tempest with > regex Andrey,   Could you give some pointers (link to patches maybe) on this. I did not completely understand what you mean here.  > > but inserting regex in plugin can restrict to have several tempest jobs > and not all gating job requires tempest > > Kind regards, > Andrey.  I think the right approach is as Andrey describes, build a custom test job appropriate for your system.  I'm not a huge fan of having plugins disable tests themselves because it's not clear to the consumer that some part of your validation stack was just disabled. I think that should be a separate specific decision made by whoever is running those tests to turn a thing off. I think the consumer is aware because he is using the enable_pluginso everything that the plugin sets or unsets is effective for this env.I feel plugin havign ability to set tempest regex also helps the pluginauthor ensure that the CI job runs well before the regex makes intothe plugin, as the CI job is voting for plugin repo (it is for glusterfs case)so its a good way to validate ur changes before running against cinderpatches. Since we don't have it today, we need to wait for project-configpatch to get the tempest regex changes in, hence need to wait forproject-config reviewers to bless .. which adds to the latency of gettinga fix inthanx,deepak,__OpenStack Development Mailing List (not for usage questions)Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribehttp://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [devstack] plugin: Giving ability for plugin repo to override tempest tests

[Deepak Shetty]

On Wed, Apr 15, 2015 at 3:59 PM, Sean Dague  wrote:

> On 04/15/2015 01:51 AM, Andrey M. Pavlov wrote:
> > Hi,
> >
> > We have similar situation with our stackforge/ec2-api project.
> > Patches in openstack repos can break our plugin.
> > But I don't see any possibility until our project not in OpenStack repo.
> > And I don't understand how plugins for tempest can help to us.
> >
> > My thoughts is only ec2-api gating jobs(this is implemented now in our
> > gating):
> > - make own tempest job with plugin definition and restrict tempest with
> > regex
>

Andrey,
   Could you give some pointers (link to patches maybe) on this. I did not
completely understand what you mean here.


> >
> > but inserting regex in plugin can restrict to have several tempest jobs
> > and not all gating job requires tempest
> >
> > Kind regards,
> > Andrey.
>
> I think the right approach is as Andrey describes, build a custom test
> job appropriate for your system.
>
> I'm not a huge fan of having plugins disable tests themselves because
> it's not clear to the consumer that some part of your validation stack
> was just disabled. I think that should be a separate specific decision
> made by whoever is running those tests to turn a thing off.
>

I think the consumer is aware because he is using the enable_plugin
so everything that the plugin sets or unsets is effective for this env.

I feel plugin havign ability to set tempest regex also helps the plugin
author ensure that the CI job runs well before the regex makes into
the plugin, as the CI job is voting for plugin repo (it is for glusterfs
case)

so its a good way to validate ur changes before running against cinder
patches. Since we don't have it today, we need to wait for project-config
patch to get the tempest regex changes in, hence need to wait for
project-config reviewers to bless .. which adds to the latency of getting
a fix in

thanx,
deepak
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [devstack] plugin: Giving ability for plugin repo to override tempest tests

[Deepak Shetty]

On Wed, Apr 15, 2015 at 3:23 PM, Andrey M. Pavlov 
wrote:

>
> And another solution - make tempest job as non-voting for your
> project(glusterfs) like we did it for ec2-api.
>
> Andrey.
>

We already do that for glusterfs & its voting,  but changes to tempest
regex doesn't go thru the plugin repo, so can't validate it until the
project-config patch is merged

thanx,
deepak
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [devstack] plugin: Giving ability for plugin repo to override tempest tests

[Sean Dague]

On 04/15/2015 01:51 AM, Andrey M. Pavlov wrote:
> Hi,
>  
> We have similar situation with our stackforge/ec2-api project.
> Patches in openstack repos can break our plugin.
> But I don't see any possibility until our project not in OpenStack repo.
> And I don't understand how plugins for tempest can help to us.
>  
> My thoughts is only ec2-api gating jobs(this is implemented now in our
> gating):
> - make own tempest job with plugin definition and restrict tempest with
> regex 
>  
> but inserting regex in plugin can restrict to have several tempest jobs
> and not all gating job requires tempest
>  
> Kind regards,
> Andrey.

I think the right approach is as Andrey describes, build a custom test
job appropriate for your system.

I'm not a huge fan of having plugins disable tests themselves because
it's not clear to the consumer that some part of your validation stack
was just disabled. I think that should be a separate specific decision
made by whoever is running those tests to turn a thing off.

-Sean

-- 
Sean Dague
http://dague.net

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [devstack] plugin: Giving ability for plugin repo to override tempest tests

[Andrey M . Pavlov]

 And another solution - make tempest job as non-voting for your project(glusterfs) like we did it for ec2-api. Andrey. 15.04.2015, 08:30, "Deepak Shetty" :Forgot to mention the patch links.. see belowOn Wed, Apr 15, 2015 at 10:49 AM, Deepak Shetty  wrote:Hi Stackers,   devstack-plugin doesn't have ability to override tempest settings. Currently it onlyprovides a way to override localrc settings, but over time we have seen that many CI jobsfail due to many different issues (lack of support, new tempest test enabled recently which fails for a particular backend etc) and there is no way for the plugin to override tempest tests usingits stackforge/{plugin} repo   Recently GlusterFS hit upon a issue due to enablement of test_volume_boot_pattern testfor which we needed to skip this test (until the issue gets fixed) and sent patch [1] but it was shot down (see [1]'s comments) and hence we sent another patch to disable it using tempest regex filter to project-config [2] ) Since devstack-plugin uses common template for all 3rd party CI jobs, it becomes a bit messy to skip for backend specific only.  It would good and ideal to have plugin be able to do the same via stackforge/{plugin} repothat ways the CI job can be tested with the tempest changes well before it run against cinder patches.  Something liek tempest/settings in plugin repo and change devstack-gate-wrap.sh ? to source tempest regex from plugin repo and pass on to the tempest cmdline ?Thoughts ?[1]: https://review.openstack.org/172841[2]: https://review.openstack.org/173408,__OpenStack Development Mailing List (not for usage questions)Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribehttp://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [devstack] plugin: Giving ability for plugin repo to override tempest tests

[Andrey M . Pavlov]

Hi, We have similar situation with our stackforge/ec2-api project.Patches in openstack repos can break our plugin.But I don't see any possibility until our project not in OpenStack repo.And I don't understand how plugins for tempest can help to us. My thoughts is only ec2-api gating jobs(this is implemented now in our gating):- make own tempest job with plugin definition and restrict tempest with regex  but inserting regex in plugin can restrict to have several tempest jobsand not all gating job requires tempest Kind regards,Andrey. 15.04.2015, 08:24, "Deepak Shetty" :Hi Stackers,   devstack-plugin doesn't have ability to override tempest settings. Currently it onlyprovides a way to override localrc settings, but over time we have seen that many CI jobsfail due to many different issues (lack of support, new tempest test enabled recently which fails for a particular backend etc) and there is no way for the plugin to override tempest tests usingits stackforge/{plugin} repo   Recently GlusterFS hit upon a issue due to enablement of test_volume_boot_pattern testfor which we needed to skip this test (until the issue gets fixed) and sent patch [1] but it was shot down (see [1]'s comments) and hence we sent another patch to disable it using tempest regex filter to project-config [2] ) Since devstack-plugin uses common template for all 3rd party CI jobs, it becomes a bit messy to skip for backend specific only.  It would good and ideal to have plugin be able to do the same via stackforge/{plugin} repothat ways the CI job can be tested with the tempest changes well before it run against cinder patches.  Something liek tempest/settings in plugin repo and change devstack-gate-wrap.sh ? to source tempest regex from plugin repo and pass on to the tempest cmdline ?Thoughts ?thanx,deepak,__OpenStack Development Mailing List (not for usage questions)Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribehttp://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [devstack] plugin: Giving ability for plugin repo to override tempest tests

[Deepak Shetty]

Forgot to mention the patch links.. see below


On Wed, Apr 15, 2015 at 10:49 AM, Deepak Shetty  wrote:

> Hi Stackers,
>devstack-plugin doesn't have ability to override tempest settings.
> Currently it only
> provides a way to override localrc settings, but over time we have seen
> that many CI jobs
> fail due to many different issues (lack of support, new tempest test
> enabled recently which fails for a particular backend etc) and there is no
> way for the plugin to override tempest tests using
> its stackforge/{plugin} repo
>
>Recently GlusterFS hit upon a issue due to enablement of
> test_volume_boot_pattern test
> for which we needed to skip this test (until the issue gets fixed) and
> sent patch [1] but it was shot down (see [1]'s comments) and hence we sent
> another patch to disable it using tempest regex filter to project-config
> [2] ) Since devstack-plugin uses common template for all 3rd party CI jobs,
> it becomes a bit messy to skip for backend specific only.
>
>   It would good and ideal to have plugin be able to do the same via
> stackforge/{plugin} repo
> that ways the CI job can be tested with the tempest changes well before it
> run against cinder patches.  Something liek tempest/settings in plugin repo
> and change devstack-gate-wrap.sh ? to source tempest regex from plugin repo
> and pass on to the tempest cmdline ?
>
> Thoughts ?
>

[1]: https://review.openstack.org/172841
[2]: https://review.openstack.org/173408
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [devstack] plugin: Giving ability for plugin repo to override tempest tests

[Deepak Shetty]

Hi Stackers,
   devstack-plugin doesn't have ability to override tempest settings.
Currently it only
provides a way to override localrc settings, but over time we have seen
that many CI jobs
fail due to many different issues (lack of support, new tempest test
enabled recently which fails for a particular backend etc) and there is no
way for the plugin to override tempest tests using
its stackforge/{plugin} repo

   Recently GlusterFS hit upon a issue due to enablement of
test_volume_boot_pattern test
for which we needed to skip this test (until the issue gets fixed) and sent
patch [1] but it was shot down (see [1]'s comments) and hence we sent
another patch to disable it using tempest regex filter to project-config
[2] ) Since devstack-plugin uses common template for all 3rd party CI jobs,
it becomes a bit messy to skip for backend specific only.

  It would good and ideal to have plugin be able to do the same via
stackforge/{plugin} repo
that ways the CI job can be tested with the tempest changes well before it
run against cinder patches.  Something liek tempest/settings in plugin repo
and change devstack-gate-wrap.sh ? to source tempest regex from plugin repo
and pass on to the tempest cmdline ?

Thoughts ?

thanx,
deepak
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev