Re: [openstack-dev] Ironic integration CI jobs

[Julia Kreger]

On Wed, Oct 31, 2018 at 7:38 AM Dmitry Tantsur  wrote:


> [trim]
>


> > Ditto, grenade jobs do not cover our tests at all. Also this is the
> very job we
> > run on other projects (nova, neutron, maybe more), so it will be a
> bit painful
> > to remove it.
> >
> >
> > We run the basic baremetal ops test, which tests deploy. If we're
> already
> > covering the same code paths in other tests (which I feel we are), then
> the test
> > feels redundant to me. I'm not worried about the effort to change the
> job in
> > other gates. We really need to pull agent_ipmitool out of the name if we
> keep it
> > anyway... which still means going through zuul configs.
>
> Do not smoke tests cover rescue with bare metal? Because our jobs do.
>
>
Smoke tests do not as far as I can tell, but I believe we run rescue by
default when our test scenarios execute on our other tempest executing jobs
as well as it is a superset of the main scenario.
Random example testr results:
http://logs.openstack.org/72/614472/1/check/ironic-tempest-dsvm-ipa-partition-redfish-tinyipa/7537b02/testr_results.html.gz
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Ironic integration CI jobs

[Dmitry Tantsur]

On 10/31/18 2:57 PM, Julia Kreger wrote:



On Wed, Oct 31, 2018 at 5:44 AM Dmitry Tantsur > wrote:


Hi,

On 10/31/18 1:36 AM, Julia Kreger wrote:
[trim]
 >
 > ironic-tempest-dsvm-ipa-wholedisk-agent_ipmitool-tinyipa-multinode - This
job is
 > essentially the same as our grenade mutlinode job, the only difference 
being
 > grenade.

Nope, not the same. Grenade jobs run only smoke tests, this job runs

https://github.com/openstack/ironic-tempest-plugin/blob/master/ironic_tempest_plugin/tests/scenario/test_baremetal_multitenancy.py


Ugh, Looking closer, we still end up deploying when the smoke tests run. It 
feels like the only real difference between what is being exercised is that one 
our explicit test scenario of putting two instances on two separate networks and 
validating connectivity is not present between the two. I guess I'm failing to 
see why we need all of the setup and infrastructure when we're just testing 
pluggable network bits and settings their upon. Maybe it is a good cantidate for 
looking at evolving how we handle scenario testing so we reduce our gate load 
and resulting wait for test results.


 > ironic-tempest-dsvm-ipa-wholedisk-bios-agent_ipmitool-tinyipa - This job
 > essentially just duplicates the functionality already covered in other 
jobs,
 > including the grenade job.

Ditto, grenade jobs do not cover our tests at all. Also this is the very 
job we
run on other projects (nova, neutron, maybe more), so it will be a bit 
painful
to remove it.


We run the basic baremetal ops test, which tests deploy. If we're already 
covering the same code paths in other tests (which I feel we are), then the test 
feels redundant to me. I'm not worried about the effort to change the job in 
other gates. We really need to pull agent_ipmitool out of the name if we keep it 
anyway... which still means going through zuul configs.


Do not smoke tests cover rescue with bare metal? Because our jobs do.



[trim]


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Ironic integration CI jobs

[Julia Kreger]

On Wed, Oct 31, 2018 at 5:44 AM Dmitry Tantsur  wrote:

> Hi,
>
> On 10/31/18 1:36 AM, Julia Kreger wrote:
> [trim]
> >
> > ironic-tempest-dsvm-ipa-wholedisk-agent_ipmitool-tinyipa-multinode -
> This job is
> > essentially the same as our grenade mutlinode job, the only difference
> being
> > grenade.
>
> Nope, not the same. Grenade jobs run only smoke tests, this job runs
>
> https://github.com/openstack/ironic-tempest-plugin/blob/master/ironic_tempest_plugin/tests/scenario/test_baremetal_multitenancy.py
>

Ugh, Looking closer, we still end up deploying when the smoke tests run. It
feels like the only real difference between what is being exercised is that
one our explicit test scenario of putting two instances on two separate
networks and validating connectivity is not present between the two. I
guess I'm failing to see why we need all of the setup and infrastructure
when we're just testing pluggable network bits and settings their upon.
Maybe it is a good cantidate for looking at evolving how we handle scenario
testing so we reduce our gate load and resulting wait for test results.

> ironic-tempest-dsvm-ipa-wholedisk-bios-agent_ipmitool-tinyipa - This job
> > essentially just duplicates the functionality already covered in other
> jobs,
> > including the grenade job.
>
> Ditto, grenade jobs do not cover our tests at all. Also this is the very
> job we
> run on other projects (nova, neutron, maybe more), so it will be a bit
> painful
> to remove it.
>

We run the basic baremetal ops test, which tests deploy. If we're already
covering the same code paths in other tests (which I feel we are), then the
test feels redundant to me. I'm not worried about the effort to change the
job in other gates. We really need to pull agent_ipmitool out of the name
if we keep it anyway... which still means going through zuul configs.

> [trim]
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Ironic integration CI jobs

[Dmitry Tantsur]

Hi,

On 10/31/18 1:36 AM, Julia Kreger wrote:
With the discussion of CI jobs and the fact that I have been finding myself 
checking job status several times a day so early in the cycle, I think it is 
time for ironic to revisit many of our CI jobs.


The bottom line is ironic is very resource intensive to test. A lot of that is 
because of the underlying way we enroll/manage nodes and then execute the 
integration scenarios emulating bare metal. I think we can improve that with 
some ansible.


In the mean time I created a quick chart[1] to try and make sense out overall 
integration coverage and I think it makes sense to remove three of the jobs.


ironic-tempest-dsvm-ipa-wholedisk-agent_ipmitool-tinyipa-multinode - This job is 
essentially the same as our grenade mutlinode job, the only difference being 
grenade.


Nope, not the same. Grenade jobs run only smoke tests, this job runs 
https://github.com/openstack/ironic-tempest-plugin/blob/master/ironic_tempest_plugin/tests/scenario/test_baremetal_multitenancy.py


ironic-tempest-dsvm-ipa-wholedisk-bios-agent_ipmitool-tinyipa - This job 
essentially just duplicates the functionality already covered in other jobs, 
including the grenade job.


Ditto, grenade jobs do not cover our tests at all. Also this is the very job we 
run on other projects (nova, neutron, maybe more), so it will be a bit painful 
to remove it.


ironic-tempest-dsvm-bfv - This presently non-voting job validates that the iPXE 
mode of the 'pxe' boot interface supports boot from volume. It was superseded by 
ironic-tempest-dsvm-ipxe-bfv which focuses on the use of the 'ipxe' boot 
interface. The underlying code is all the same deep down in all of the helper 
methods.


+1 to this.

Dmitry



I'll go ahead and put this up as a topic for our weekly meeting next week so we 
can discuss.


Thanks,

-Julia

[1]: https://ethercalc.openstack.org/ces0z3xjb1ir

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Ironic integration CI jobs

[Nate Johnston]

On Tue, Oct 30, 2018 at 05:36:09PM -0700, Julia Kreger wrote:

> ironic-tempest-dsvm-ipa-wholedisk-bios-agent_ipmitool-tinyipa - This job
> essentially just duplicates the functionality already covered in other
> jobs, including the grenade job.

FYI this job runs in the Neutron check queue (non-voting).  Would you
like to substitute a different Ironic job to run on all neutron changes,
or should that requirement be revisited?

Thanks!

Nate

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] Ironic integration CI jobs

[Julia Kreger]

With the discussion of CI jobs and the fact that I have been finding myself
checking job status several times a day so early in the cycle, I think it
is time for ironic to revisit many of our CI jobs.

The bottom line is ironic is very resource intensive to test. A lot of that
is because of the underlying way we enroll/manage nodes and then execute
the integration scenarios emulating bare metal. I think we can improve that
with some ansible.

In the mean time I created a quick chart[1] to try and make sense out
overall integration coverage and I think it makes sense to remove three of
the jobs.

ironic-tempest-dsvm-ipa-wholedisk-agent_ipmitool-tinyipa-multinode - This
job is essentially the same as our grenade mutlinode job, the only
difference being grenade.
ironic-tempest-dsvm-ipa-wholedisk-bios-agent_ipmitool-tinyipa - This job
essentially just duplicates the functionality already covered in other
jobs, including the grenade job.
ironic-tempest-dsvm-bfv - This presently non-voting job validates that the
iPXE mode of the 'pxe' boot interface supports boot from volume. It was
superseded by ironic-tempest-dsvm-ipxe-bfv which focuses on the use of the
'ipxe' boot interface. The underlying code is all the same deep down in all
of the helper methods.

I'll go ahead and put this up as a topic for our weekly meeting next week
so we can discuss.

Thanks,

-Julia

[1]: https://ethercalc.openstack.org/ces0z3xjb1ir
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev