Re: [openstack-dev] [qa] Tempest pre-provisioned credentials in the gate

2016-06-16 Thread Ken'ichi Ohmichi 
2016-06-14 17:00 GMT-07:00 Andrea Frittoli :
> Dear all,
>
> TL;DR: I'd like to propose to start running some of the existing dsvm
> check/gate jobs using Tempest pre-provisioned credentials.
>
> Full Text:
> Tempest provides tests with two mechanisms to acquire test credentials [0]:
> dynamic credentials and pre-provisioned ones.
>
> The current check and gate jobs only use the dynamic credentials provider.
>
> The pre-provisioned credentials provider has been introduced to support
> running test in parallel without the need of having access to admin
> credentials in tempest configuration file - which is a valid use case
> especially when testing public clouds or in general a deployment that is not
> own by who runs the test.
>
> As a small extra, since pre-provisioned credentials are re-used to run many
> tests during a CI test run, they give an opportunity to discover issues
> related to cleanup of test resources.

This is a significant merit for Tempest development.
So +1 for enabling the credential on the gate.

> Pre-provisioned credentials is currently used in periodic jobs [1][2] - as
> well as an experimental job defined for tempest changes. This means that
> even if we are careful, there is a good chance for it to be inadvertently
> broken by a change.
>
> Until recently the periodic job suffered a racy failure on object-storage
> tests. A recent refactor [3] of the tool that pre-proprovisioned the
> accounts has fixed the issue: the past 8 runs of the periodic jobs have not
> encountered that race anymore [4][5].
>
> Specifically I'd like to propose changing to start changing of the neutron
> jobs [6].

I am not sure why don't we add another simple job(non neutron job)
which just enables this.
But maybe it is better to discuss which job is nice for enabling
pre-provisioned cred on the review.

Thanks
Ken Ohmichi

---

> [0]
> http://docs.openstack.org/developer/tempest/configuration.html#credential-provider-mechanisms
> [1]
> http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/jobs/devstack-gate.yaml#n220
> [2]
> http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/jobs/devstack-gate.yaml#n253
> [3] https://review.openstack.org/#/c/317105/
> [4]
> http://status.openstack.org/openstack-health/#/job/periodic-tempest-dsvm-full-test-accounts-master
> [5]
> http://status.openstack.org/openstack-health/#/job/periodic-tempest-dsvm-neutron-full-test-accounts-master
> [6] https://review.openstack.org/329723
>
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [qa] Tempest pre-provisioned credentials in the gate

2016-06-15 Thread Ghanshyam Mann 
Big +1 for that which makes pre-provisioned cred testing more thoroughly.

And at the end we should divide the existing jobs among dynamic and 
pre-provisioned account so that we would not leave any of them less tested.

Thanks
gmann
From: Andrea Frittoli [mailto:andrea.fritt...@gmail.com]
Sent: 15 June 2016 09:01
To: OpenStack Development Mailing List (not for usage questions) 
<openstack-dev@lists.openstack.org>
Subject: [openstack-dev] [qa] Tempest pre-provisioned credentials in the gate

Dear all,

TL;DR: I'd like to propose to start running some of the existing dsvm 
check/gate jobs using Tempest pre-provisioned credentials.

Full Text:
Tempest provides tests with two mechanisms to acquire test credentials [0]: 
dynamic credentials and pre-provisioned ones.

The current check and gate jobs only use the dynamic credentials provider.

The pre-provisioned credentials provider has been introduced to support running 
test in parallel without the need of having access to admin credentials in 
tempest configuration file - which is a valid use case especially when testing 
public clouds or in general a deployment that is not own by who runs the test.

As a small extra, since pre-provisioned credentials are re-used to run many 
tests during a CI test run, they give an opportunity to discover issues related 
to cleanup of test resources.

Pre-provisioned credentials is currently used in periodic jobs [1][2] - as well 
as an experimental job defined for tempest changes. This means that even if we 
are careful, there is a good chance for it to be inadvertently broken by a 
change.

Until recently the periodic job suffered a racy failure on object-storage 
tests. A recent refactor [3] of the tool that pre-proprovisioned the accounts 
has fixed the issue: the past 8 runs of the periodic jobs have not encountered 
that race anymore [4][5].

Specifically I'd like to propose changing to start changing of the neutron jobs 
[6].

Andrea Frittoli

[0] 
http://docs.openstack.org/developer/tempest/configuration.html#credential-provider-mechanisms
[1] 
http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/jobs/devstack-gate.yaml#n220
[2] 
http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/jobs/devstack-gate.yaml#n253
[3] https://review.openstack.org/#/c/317105/
[4] 
http://status.openstack.org/openstack-health/#/job/periodic-tempest-dsvm-full-test-accounts-master
[5] 
http://status.openstack.org/openstack-health/#/job/periodic-tempest-dsvm-neutron-full-test-accounts-master
[6] https://review.openstack.org/329723


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [qa] Tempest pre-provisioned credentials in the gate

2016-06-15 Thread Jordan Pittier 
On Wed, Jun 15, 2016 at 2:00 AM, Andrea Frittoli 
wrote:

> Dear all,
>
> TL;DR: I'd like to propose to start running some of the existing dsvm
> check/gate jobs using Tempest pre-provisioned credentials.
>
> Full Text:
> Tempest provides tests with two mechanisms to acquire test credentials
> [0]: dynamic credentials and pre-provisioned ones.
>
> The current check and gate jobs only use the dynamic credentials provider.
>
> The pre-provisioned credentials provider has been introduced to support
> running test in parallel without the need of having access to admin
> credentials in tempest configuration file - which is a valid use case
> especially when testing public clouds or in general a deployment that is
> not own by who runs the test.
>
> As a small extra, since pre-provisioned credentials are re-used to run
> many tests during a CI test run, they give an opportunity to discover
> issues related to cleanup of test resources.
>
> Pre-provisioned credentials is currently used in periodic jobs [1][2] - as
> well as an experimental job defined for tempest changes. This means that
> even if we are careful, there is a good chance for it to be inadvertently
> broken by a change.
>
> Until recently the periodic job suffered a racy failure on object-storage
> tests. A recent refactor [3] of the tool that pre-proprovisioned the
> accounts has fixed the issue: the past 8 runs of the periodic jobs have not
> encountered that race anymore [4][5].
>
> Specifically I'd like to propose changing to start changing of the neutron
> jobs [6].
>
> Andrea Frittoli
>
> [0]
> http://docs.openstack.org/developer/tempest/configuration.html#credential-provider-mechanisms
> [1]
> http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/jobs/devstack-gate.yaml#n220
> [2]
> http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/jobs/devstack-gate.yaml#n253
> [3] https://review.openstack.org/#/c/317105/
> [4]
> http://status.openstack.org/openstack-health/#/job/periodic-tempest-dsvm-full-test-accounts-master
> [5]
> http://status.openstack.org/openstack-health/#/job/periodic-tempest-dsvm-neutron-full-test-accounts-master
> [6] https://review.openstack.org/329723
>
>
I like the idea. If the jobs with the preprov credentials are as stable as
the ones running dynamic credentials, that's fine. Also I like that we
don"t introduce new jobs but instead changing the configuration of existing
jobs: we already have a lot of jobs in Tempest.

In my previous job, we also used Tempest to validate our cloud deployment
and we didn"t have the admin credentials, so yeah, preprov credentials are
useful.

-- 
 
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [qa] Tempest pre-provisioned credentials in the gate

2016-06-14 Thread Andrea Frittoli 
Dear all,

TL;DR: I'd like to propose to start running some of the existing dsvm
check/gate jobs using Tempest pre-provisioned credentials.

Full Text:
Tempest provides tests with two mechanisms to acquire test credentials [0]:
dynamic credentials and pre-provisioned ones.

The current check and gate jobs only use the dynamic credentials provider.

The pre-provisioned credentials provider has been introduced to support
running test in parallel without the need of having access to admin
credentials in tempest configuration file - which is a valid use case
especially when testing public clouds or in general a deployment that is
not own by who runs the test.

As a small extra, since pre-provisioned credentials are re-used to run many
tests during a CI test run, they give an opportunity to discover issues
related to cleanup of test resources.

Pre-provisioned credentials is currently used in periodic jobs [1][2] - as
well as an experimental job defined for tempest changes. This means that
even if we are careful, there is a good chance for it to be inadvertently
broken by a change.

Until recently the periodic job suffered a racy failure on object-storage
tests. A recent refactor [3] of the tool that pre-proprovisioned the
accounts has fixed the issue: the past 8 runs of the periodic jobs have not
encountered that race anymore [4][5].

Specifically I'd like to propose changing to start changing of the neutron
jobs [6].

Andrea Frittoli

[0]
http://docs.openstack.org/developer/tempest/configuration.html#credential-provider-mechanisms
[1]
http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/jobs/devstack-gate.yaml#n220
[2]
http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/jobs/devstack-gate.yaml#n253
[3] https://review.openstack.org/#/c/317105/
[4]
http://status.openstack.org/openstack-health/#/job/periodic-tempest-dsvm-full-test-accounts-master
[5]
http://status.openstack.org/openstack-health/#/job/periodic-tempest-dsvm-neutron-full-test-accounts-master
[6] https://review.openstack.org/329723
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev