Re: [openstack-dev] [Magnum][Testing] Reduce Functional testing on gate.

2015-11-12 Thread Egor Guz 
Eli,

First of all I would like to say thank you for your effort (I never seen so 
many path sets ;)), but I don’t think we should remove “tls_disabled=True” 
tests from gates now (maybe in L).
It’s still vey commonly used feature and backup plan if TLS doesn’t work for 
some reasons.

I think it’s good idea to group tests per pipeline we should definitely follow 
it.

—
Egor

From: "Qiao,Liyong" <liyong.q...@intel.com<mailto:liyong.q...@intel.com>>
Reply-To: "OpenStack Development Mailing List (not for usage questions)" 
<openstack-dev@lists.openstack.org<mailto:openstack-dev@lists.openstack.org>>
Date: Wednesday, November 11, 2015 at 23:02
To: 
"openstack-dev@lists.openstack.org<mailto:openstack-dev@lists.openstack.org>" 
<openstack-dev@lists.openstack.org<mailto:openstack-dev@lists.openstack.org>>
Subject: [openstack-dev] [Magnum][Testing] Reduce Functional testing on gate.

hello all:

I will update some Magnum functional testing status, functional/integration 
testing
is important to us, since we change/modify the Heat template rapidly, we need to
verify the modification is correct, so we need to cover all templates Magnum 
has.
and currently we only has k8s testing(only test with atomic image), we need to
add more, like swarm(WIP), mesos(under plan), also , we may need to support COS 
image.
lots of work need to be done.

for the functional testing time costing, we discussed during the Tokyo summit,
Adrian expected that we can reduce the timing cost to 20min.

I did some analyses on the functional/integrated testing on gate pipeline.
the stages will be follows:
take k8s functional testing for example, we did follow testing case:

1) baymodel creation
2) bay(tls_disabled=True) creation/deletion
3) bay(tls_disabled=False) creation to testing k8s api and delete it after 
testing.

for each stage, the time costing is follows:

  *   devstack prepare: 5-6 mins
  *   Running devstack: 15 mins(include downloading atomic image)
  *   1) and 2) 15 mins
  *   3) 15 +3 mins

totally about 60 mins currently a example is 1h 05m 57s
see 
http://logs.openstack.org/10/243910/1/check/gate-functional-dsvm-magnum-k8s/5e61039/console.html
for all time stamps.

I don't think it is possible to reduce time to 20 mins, since devstack setup 
will take 20 mins already.

To reduce time, I suggest to only create 1 bay each pipeline and do vary kinds 
of testing
on this bay, if want to test some specify bay (for example, network_driver 
etc), create
a new pipeline .

So, Ｉ think we can delete 2), since 3) will do similar things(create/delete), 
the different is
3) use tls_disabled=False. what do you think ?
see https://review.openstack.org/244378 for the time costing, will reduce to 45 
min (48m 50s in the example.)

=
For other related functional testing works:
I 'v done the split of functional testing per COE, we have pipeline as:

  *   gate-functional-dsvm-magnum-api 30 mins
  *   gate-functional-dsvm-magnum-k8s 60 mins

And for swam pipeline, patches is done, under reviewing now(works fine on gate)
https://review.openstack.org/244391
https://review.openstack.org/226125



--
BR, Eli(Li Yong)Qiao
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Magnum][Testing] Reduce Functional testing on gate.

2015-11-12 Thread Ton Ngo 

Thanks Eli for the analysis.  I notice that the time to download the image
is only around 1:15 mins out of some 21 mins to set up devstack.  So it
seems trying to reduce the size of the image won't make a significant
improvement in the devstack time.   I wonder how the image size affects the
VM creation time for the cluster.  If we can look at the Heat event stream,
we might get an idea.
Ton,




From:   Egor Guz <e...@walmartlabs.com>
To: "openstack-dev@lists.openstack.org"
<openstack-dev@lists.openstack.org>
Date:   11/12/2015 05:25 PM
Subject:        Re: [openstack-dev] [Magnum][Testing] Reduce Functional testing
on gate.



Eli,

First of all I would like to say thank you for your effort (I never seen so
many path sets ;)), but I don’t think we should remove “tls_disabled=True”
tests from gates now (maybe in L).
It’s still vey commonly used feature and backup plan if TLS doesn’t work
for some reasons.

I think it’s good idea to group tests per pipeline we should definitely
follow it.

?
Egor

From: "Qiao,Liyong" <liyong.q...@intel.com<mailto:liyong.q...@intel.com>>
Reply-To: "OpenStack Development Mailing List (not for usage questions)"
<openstack-dev@lists.openstack.org<mailto:openstack-dev@lists.openstack.org
>>
Date: Wednesday, November 11, 2015 at 23:02
To: "openstack-dev@lists.openstack.org<
mailto:openstack-dev@lists.openstack.org>"
<openstack-dev@lists.openstack.org<mailto:openstack-dev@lists.openstack.org
>>
Subject: [openstack-dev] [Magnum][Testing] Reduce Functional testing on
gate.

hello all:

I will update some Magnum functional testing status, functional/integration
testing
is important to us, since we change/modify the Heat template rapidly, we
need to
verify the modification is correct, so we need to cover all templates
Magnum has.
and currently we only has k8s testing(only test with atomic image), we need
to
add more, like swarm(WIP), mesos(under plan), also , we may need to support
COS image.
lots of work need to be done.

for the functional testing time costing, we discussed during the Tokyo
summit,
Adrian expected that we can reduce the timing cost to 20min.

I did some analyses on the functional/integrated testing on gate pipeline.
the stages will be follows:
take k8s functional testing for example, we did follow testing case:

1) baymodel creation
2) bay(tls_disabled=True) creation/deletion
3) bay(tls_disabled=False) creation to testing k8s api and delete it after
testing.

for each stage, the time costing is follows:

  *   devstack prepare: 5-6 mins
  *   Running devstack: 15 mins(include downloading atomic image)
  *   1) and 2) 15 mins
  *   3) 15 +3 mins

totally about 60 mins currently a example is 1h 05m 57s
see
http://logs.openstack.org/10/243910/1/check/gate-functional-dsvm-magnum-k8s/5e61039/console.html

for all time stamps.

I don't think it is possible to reduce time to 20 mins, since devstack
setup will take 20 mins already.

To reduce time, I suggest to only create 1 bay each pipeline and do vary
kinds of testing
on this bay, if want to test some specify bay (for example, network_driver
etc), create
a new pipeline .

So, Ｉ think we can delete 2), since 3) will do similar things
(create/delete), the different is
3) use tls_disabled=False. what do you think ?
see https://review.openstack.org/244378 for the time costing, will reduce
to 45 min (48m 50s in the example.)

=
For other related functional testing works:
I 'v done the split of functional testing per COE, we have pipeline as:

  *   gate-functional-dsvm-magnum-api 30 mins
  *   gate-functional-dsvm-magnum-k8s 60 mins

And for swam pipeline, patches is done, under reviewing now(works fine on
gate)
https://review.openstack.org/244391
https://review.openstack.org/226125



--
BR, Eli(Li Yong)Qiao
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Magnum][Testing] Reduce Functional testing on gate.

2015-11-11 Thread Qiao,Liyong 

hello all:

I will update some Magnum functional testing status, 
functional/integration testing
is important to us, since we change/modify the Heat template rapidly, we 
need to
verify the modification is correct, so we need to cover all templates 
Magnum has.
and currently we only has k8s testing(only test with atomic image), we 
need to
add more, like swarm(WIP), mesos(under plan), also , we may need to 
support COS image.

lots of work need to be done.

for the functional testing time costing, we discussed during the Tokyo 
summit,

Adrian expected that we can reduce the timing cost to 20min.

I did some analyses on the functional/integrated testing on gate pipeline.
the stages will be follows:
take k8s functional testing for example, we did follow testing case:

1) baymodel creation
2) bay(tls_disabled=True) creation/deletion
3) bay(tls_disabled=False) creation to testing k8s api and delete it 
after testing.


for each stage, the time costing is follows:

 * devstack prepare: 5-6 mins
 * Running devstack: 15 mins(include downloading atomic image)
 * 1) and 2) 15 mins
 * 3) 15 +3 mins

totally about 60 mins currently a example is 1h 05m 57s
see 
http://logs.openstack.org/10/243910/1/check/gate-functional-dsvm-magnum-k8s/5e61039/console.html

for all time stamps.

I don't think it is possible to reduce time to 20 mins, since devstack 
setup will take 20 mins already.


To reduce time, I suggest to only create 1 bay each pipeline and do vary 
kinds of testing
on this bay, if want to test some specify bay (for example, 
network_driver etc), create

a new pipeline .

So, Ｉ think we can *delete 2)*, since 3) will do similar 
things(create/delete), the different is

3) use tls_disabled=False. *what do you think *?
see https://review.openstack.org/244378 for the time costing, will 
reduce to 45 min (48m 50s in the example.)


=
For other related functional testing works:
I 'v done the split of functional testing per COE, we have pipeline as:

 * gate-functional-dsvm-magnum-api 30 mins
 * gate-functional-dsvm-magnum-k8s 60 mins

And for swam pipeline, patches is done, under reviewing now(works fine 
on gate)

https://review.openstack.org/244391
https://review.openstack.org/226125



--
BR, Eli(Li Yong)Qiao

<>__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Magnum][Testing] Reduce Functional testing on gate.

2015-11-11 Thread Adrian Otto 
Eli,

I like this proposed approach. We did have a discussion with a few Stackers 
from openstack-infra in Tokyo to express our interest in using bare metal for 
gate testing. That’s still a way out, but that may be another way to speed this 
up further. A third idea would be to adjust the nova virt driver in our 
devstack image to use libvirt/lxc by default (instead of libvirt/kvm) which 
would allow for bays to be created more rapidly. This would potentially allow 
us to to perform repeated bay creations int he same pipeline in a reasonable 
timeframe.

Adrian

On Nov 11, 2015, at 11:02 PM, Qiao,Liyong 
> wrote:

hello all:

I will update some Magnum functional testing status, functional/integration 
testing
is important to us, since we change/modify the Heat template rapidly, we need to
verify the modification is correct, so we need to cover all templates Magnum 
has.
and currently we only has k8s testing(only test with atomic image), we need to
add more, like swarm(WIP), mesos(under plan), also , we may need to support COS 
image.
lots of work need to be done.

for the functional testing time costing, we discussed during the Tokyo summit,
Adrian expected that we can reduce the timing cost to 20min.

I did some analyses on the functional/integrated testing on gate pipeline.
the stages will be follows:
take k8s functional testing for example, we did follow testing case:

1) baymodel creation
2) bay(tls_disabled=True) creation/deletion
3) bay(tls_disabled=False) creation to testing k8s api and delete it after 
testing.

for each stage, the time costing is follows:

  *   devstack prepare: 5-6 mins
  *   Running devstack: 15 mins(include downloading atomic image)
  *   1) and 2) 15 mins
  *   3) 15 +3 mins

totally about 60 mins currently a example is 1h 05m 57s
see 
http://logs.openstack.org/10/243910/1/check/gate-functional-dsvm-magnum-k8s/5e61039/console.html
for all time stamps.

I don't think it is possible to reduce time to 20 mins, since devstack setup 
will take 20 mins already.

To reduce time, I suggest to only create 1 bay each pipeline and do vary kinds 
of testing
on this bay, if want to test some specify bay (for example, network_driver 
etc), create
a new pipeline .

So, Ｉ think we can delete 2), since 3) will do similar things(create/delete), 
the different is
3) use tls_disabled=False. what do you think ?
see https://review.openstack.org/244378 for the time costing, will reduce to 45 
min (48m 50s in the example.)

=
For other related functional testing works:
I 'v done the split of functional testing per COE, we have pipeline as:

  *   gate-functional-dsvm-magnum-api 30 mins
  *   gate-functional-dsvm-magnum-k8s 60 mins

And for swam pipeline, patches is done, under reviewing now(works fine on gate)
https://review.openstack.org/244391
https://review.openstack.org/226125



--
BR, Eli(Li Yong)Qiao

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: 
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev