Re: [openstack-dev] [tempest] Implementing tempest test for Keystone federation functional tests

[Rodrigo Duarte]

On Tue, Apr 5, 2016 at 11:53 AM, Minying Lu  wrote:

> Thank you for your awesome feedbacks!
>
>
>> Another option is to add those tests to keystone itself (if you are not
>>> including tests that triggers other components APIs). See
>>> https://blueprints.launchpad.net/keystone/+spec/keystone-tempest-plugin-tests
>>>
>>>
>>
>>
> knikolla and I are looking into the keystone-tempest-plugin too thanks
> Rodrigo!
>
>
>> Again though, the problem is not where the tests live but where we run
>> them. To practically run these tests we need to either add K2K testing
>> support to devstack (not sure this is appropriate) or come up with a new
>> test environment that deploys 2 keystones and federation support that we
>> can CI against in the gate. This is doable but i think something we need
>> support with from infra before worrying about tempest.
>>
>>
> We have engineers in the team that are communicating with the infra team
> on how to set up a environment that runs the federation test. We're
> thinking about creating a 2 devstack setup with the keystones configured as
> Identity provider and service provider with federation support. Meanwhile I
> can just work on writing the test in a pre-configured environment that's
> the same as the 2 devstack setup.
>

Awesome, please share your work so I can help on that front too!


>
>
>>
>>
>>>
 The fly in the ointment for this case will be CI though. For tests to
 live in
 tempest they need to be verified by a CI system before they can land.
 So to
 land the additional testing in tempest you'll have to also ensure there
 is a
 CI job setup in infra to configure the necessary environment. While I
 think
 this is a good thing to have in the long run, it's not necessarily a
 small
 undertaking.

>>>
 -Matt Treinish


 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe:
 openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


>>>
>>>
>>> --
>>> Rodrigo Duarte Sousa
>>> Senior Quality Engineer @ Red Hat
>>> MSc in Computer Science
>>> http://rodrigods.com
>>>
>>>
>>> __
>>> OpenStack Development Mailing List (not for usage questions)
>>> Unsubscribe:
>>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>>
>>
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>


-- 
Rodrigo Duarte Sousa
Senior Quality Engineer @ Red Hat
MSc in Computer Science
http://rodrigods.com
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tempest] Implementing tempest test for Keystone federation functional tests

[Minying Lu]

Thank you for your awesome feedbacks!


> Another option is to add those tests to keystone itself (if you are not
>> including tests that triggers other components APIs). See
>> https://blueprints.launchpad.net/keystone/+spec/keystone-tempest-plugin-tests
>>
>>
>
>
knikolla and I are looking into the keystone-tempest-plugin too thanks
Rodrigo!


> Again though, the problem is not where the tests live but where we run
> them. To practically run these tests we need to either add K2K testing
> support to devstack (not sure this is appropriate) or come up with a new
> test environment that deploys 2 keystones and federation support that we
> can CI against in the gate. This is doable but i think something we need
> support with from infra before worrying about tempest.
>
>
We have engineers in the team that are communicating with the infra team on
how to set up a environment that runs the federation test. We're thinking
about creating a 2 devstack setup with the keystones configured as Identity
provider and service provider with federation support. Meanwhile I can just
work on writing the test in a pre-configured environment that's the same as
the 2 devstack setup.


>
>
>>
>>> The fly in the ointment for this case will be CI though. For tests to
>>> live in
>>> tempest they need to be verified by a CI system before they can land. So
>>> to
>>> land the additional testing in tempest you'll have to also ensure there
>>> is a
>>> CI job setup in infra to configure the necessary environment. While I
>>> think
>>> this is a good thing to have in the long run, it's not necessarily a
>>> small
>>> undertaking.
>>>
>>
>>> -Matt Treinish
>>>
>>>
>>> __
>>> OpenStack Development Mailing List (not for usage questions)
>>> Unsubscribe:
>>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>>
>>
>>
>> --
>> Rodrigo Duarte Sousa
>> Senior Quality Engineer @ Red Hat
>> MSc in Computer Science
>> http://rodrigods.com
>>
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tempest] Implementing tempest test for Keystone federation functional tests

[Minying Lu]

Thank you all for your awesome feedbacks!

On Sun, Apr 3, 2016 at 9:07 PM, Jamie Lennox  wrote:

>
>
> On 2 April 2016 at 09:21, Rodrigo Duarte  wrote:
>
>>
>>
>> On Thu, Mar 31, 2016 at 1:11 PM, Matthew Treinish 
>> wrote:
>>
>>> On Thu, Mar 31, 2016 at 11:38:55AM -0400, Minying Lu wrote:
>>> > Hi all,
>>> >
>>> > I'm working on resource federation at the Massachusetts Open Cloud. We
>>> want
>>> > to implement functional test on the k2k federation, which requires
>>> > authentication with both a local keystone and a remote keystone (in a
>>> > different cloud installation). It also requires a K2K/SAML assertion
>>> > exchange with the local and remote keystones. These functions are not
>>> > implemented in the current tempest.lib.service library, so I'm adding
>>> code
>>> > to the service library.
>>> >
>>> > My question is, is it possible to adapt keystoneauth python clients?
>>> Or do
>>> > you prefer implementing it with http requests.
>>>
>>> So tempest's clients have to be completely independent. That's part of
>>> tempest's
>>> design points about testing APIs, not client implementations. If you
>>> need to add
>>> additional functionality to the tempest clients that's fine, but pulling
>>> in
>>> keystoneauth isn't really an option.
>>>
>>
>> ++
>>
>>
>>>
>>> >
>>> > And since this test requires a lot of environment set up including: 2
>>> > separate cloud installations, shibboleth, creating mapping and
>>> protocols on
>>> > remote cloud, etc. Would it be within the scope of tempest's mission?
>>>
>>> From the tempest perspective it expects the environment to be setup and
>>> already
>>> exist by the time you run the test. If it's a valid use of the API,
>>> which I'd
>>> say this is and an important one too, then I feel it's fair game to have
>>> tests
>>> for this live in tempest. We'll just have to make the configuration
>>> options
>>> around how tempest will do this very explicit to make sure the necessary
>>> environment exists before the tests are executed.
>>>
>>
>> Another option is to add those tests to keystone itself (if you are not
>> including tests that triggers other components APIs). See
>> https://blueprints.launchpad.net/keystone/+spec/keystone-tempest-plugin-tests
>>
>>
>
> Again though, the problem is not where the tests live but where we run
> them. To practically run these tests we need to either add K2K testing
> support to devstack (not sure this is appropriate) or come up with a new
> test environment that deploys 2 keystones and federation support that we
> can CI against in the gate. This is doable but i think something we need
> support with from infra before worrying about tempest.
>
>
>

>

>>> The fly in the ointment for this case will be CI though. For tests to
>>> live in
>>> tempest they need to be verified by a CI system before they can land. So
>>> to
>>> land the additional testing in tempest you'll have to also ensure there
>>> is a
>>> CI job setup in infra to configure the necessary environment. While I
>>> think
>>> this is a good thing to have in the long run, it's not necessarily a
>>> small
>>> undertaking.
>>>
>>
>>> -Matt Treinish
>>>
>>>
>>> __
>>> OpenStack Development Mailing List (not for usage questions)
>>> Unsubscribe:
>>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>>
>>
>>
>> --
>> Rodrigo Duarte Sousa
>> Senior Quality Engineer @ Red Hat
>> MSc in Computer Science
>> http://rodrigods.com
>>
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tempest] Implementing tempest test for Keystone federation functional tests

[Jamie Lennox]

On 2 April 2016 at 09:21, Rodrigo Duarte  wrote:

>
>
> On Thu, Mar 31, 2016 at 1:11 PM, Matthew Treinish 
> wrote:
>
>> On Thu, Mar 31, 2016 at 11:38:55AM -0400, Minying Lu wrote:
>> > Hi all,
>> >
>> > I'm working on resource federation at the Massachusetts Open Cloud. We
>> want
>> > to implement functional test on the k2k federation, which requires
>> > authentication with both a local keystone and a remote keystone (in a
>> > different cloud installation). It also requires a K2K/SAML assertion
>> > exchange with the local and remote keystones. These functions are not
>> > implemented in the current tempest.lib.service library, so I'm adding
>> code
>> > to the service library.
>> >
>> > My question is, is it possible to adapt keystoneauth python clients? Or
>> do
>> > you prefer implementing it with http requests.
>>
>> So tempest's clients have to be completely independent. That's part of
>> tempest's
>> design points about testing APIs, not client implementations. If you need
>> to add
>> additional functionality to the tempest clients that's fine, but pulling
>> in
>> keystoneauth isn't really an option.
>>
>
> ++
>
>
>>
>> >
>> > And since this test requires a lot of environment set up including: 2
>> > separate cloud installations, shibboleth, creating mapping and
>> protocols on
>> > remote cloud, etc. Would it be within the scope of tempest's mission?
>>
>> From the tempest perspective it expects the environment to be setup and
>> already
>> exist by the time you run the test. If it's a valid use of the API, which
>> I'd
>> say this is and an important one too, then I feel it's fair game to have
>> tests
>> for this live in tempest. We'll just have to make the configuration
>> options
>> around how tempest will do this very explicit to make sure the necessary
>> environment exists before the tests are executed.
>>
>
> Another option is to add those tests to keystone itself (if you are not
> including tests that triggers other components APIs). See
> https://blueprints.launchpad.net/keystone/+spec/keystone-tempest-plugin-tests
>
>

Again though, the problem is not where the tests live but where we run
them. To practically run these tests we need to either add K2K testing
support to devstack (not sure this is appropriate) or come up with a new
test environment that deploys 2 keystones and federation support that we
can CI against in the gate. This is doable but i think something we need
support with from infra before worrying about tempest.



>
>> The fly in the ointment for this case will be CI though. For tests to
>> live in
>> tempest they need to be verified by a CI system before they can land. So
>> to
>> land the additional testing in tempest you'll have to also ensure there
>> is a
>> CI job setup in infra to configure the necessary environment. While I
>> think
>> this is a good thing to have in the long run, it's not necessarily a small
>> undertaking.
>>
>
>> -Matt Treinish
>>
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>
>
> --
> Rodrigo Duarte Sousa
> Senior Quality Engineer @ Red Hat
> MSc in Computer Science
> http://rodrigods.com
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tempest] Implementing tempest test for Keystone federation functional tests

[Rodrigo Duarte]

On Thu, Mar 31, 2016 at 1:11 PM, Matthew Treinish 
wrote:

> On Thu, Mar 31, 2016 at 11:38:55AM -0400, Minying Lu wrote:
> > Hi all,
> >
> > I'm working on resource federation at the Massachusetts Open Cloud. We
> want
> > to implement functional test on the k2k federation, which requires
> > authentication with both a local keystone and a remote keystone (in a
> > different cloud installation). It also requires a K2K/SAML assertion
> > exchange with the local and remote keystones. These functions are not
> > implemented in the current tempest.lib.service library, so I'm adding
> code
> > to the service library.
> >
> > My question is, is it possible to adapt keystoneauth python clients? Or
> do
> > you prefer implementing it with http requests.
>
> So tempest's clients have to be completely independent. That's part of
> tempest's
> design points about testing APIs, not client implementations. If you need
> to add
> additional functionality to the tempest clients that's fine, but pulling in
> keystoneauth isn't really an option.
>

++


>
> >
> > And since this test requires a lot of environment set up including: 2
> > separate cloud installations, shibboleth, creating mapping and protocols
> on
> > remote cloud, etc. Would it be within the scope of tempest's mission?
>
> From the tempest perspective it expects the environment to be setup and
> already
> exist by the time you run the test. If it's a valid use of the API, which
> I'd
> say this is and an important one too, then I feel it's fair game to have
> tests
> for this live in tempest. We'll just have to make the configuration options
> around how tempest will do this very explicit to make sure the necessary
> environment exists before the tests are executed.
>

Another option is to add those tests to keystone itself (if you are not
including tests that triggers other components APIs). See
https://blueprints.launchpad.net/keystone/+spec/keystone-tempest-plugin-tests


>
> The fly in the ointment for this case will be CI though. For tests to live
> in
> tempest they need to be verified by a CI system before they can land. So to
> land the additional testing in tempest you'll have to also ensure there is
> a
> CI job setup in infra to configure the necessary environment. While I think
> this is a good thing to have in the long run, it's not necessarily a small
> undertaking.
>

> -Matt Treinish
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>


-- 
Rodrigo Duarte Sousa
Senior Quality Engineer @ Red Hat
MSc in Computer Science
http://rodrigods.com
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tempest] Implementing tempest test for Keystone federation functional tests

[Matthew Treinish]

On Thu, Mar 31, 2016 at 11:38:55AM -0400, Minying Lu wrote:
> Hi all,
> 
> I'm working on resource federation at the Massachusetts Open Cloud. We want
> to implement functional test on the k2k federation, which requires
> authentication with both a local keystone and a remote keystone (in a
> different cloud installation). It also requires a K2K/SAML assertion
> exchange with the local and remote keystones. These functions are not
> implemented in the current tempest.lib.service library, so I'm adding code
> to the service library.
> 
> My question is, is it possible to adapt keystoneauth python clients? Or do
> you prefer implementing it with http requests.

So tempest's clients have to be completely independent. That's part of tempest's
design points about testing APIs, not client implementations. If you need to add
additional functionality to the tempest clients that's fine, but pulling in
keystoneauth isn't really an option.

> 
> And since this test requires a lot of environment set up including: 2
> separate cloud installations, shibboleth, creating mapping and protocols on
> remote cloud, etc. Would it be within the scope of tempest's mission?

From the tempest perspective it expects the environment to be setup and already
exist by the time you run the test. If it's a valid use of the API, which I'd
say this is and an important one too, then I feel it's fair game to have tests
for this live in tempest. We'll just have to make the configuration options
around how tempest will do this very explicit to make sure the necessary
environment exists before the tests are executed.

The fly in the ointment for this case will be CI though. For tests to live in
tempest they need to be verified by a CI system before they can land. So to
land the additional testing in tempest you'll have to also ensure there is a
CI job setup in infra to configure the necessary environment. While I think
this is a good thing to have in the long run, it's not necessarily a small
undertaking.

-Matt Treinish


signature.asc
Description: PGP signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [tempest] Implementing tempest test for Keystone federation functional tests

[Minying Lu]

Hi all,

I'm working on resource federation at the Massachusetts Open Cloud. We want
to implement functional test on the k2k federation, which requires
authentication with both a local keystone and a remote keystone (in a
different cloud installation). It also requires a K2K/SAML assertion
exchange with the local and remote keystones. These functions are not
implemented in the current tempest.lib.service library, so I'm adding code
to the service library.

My question is, is it possible to adapt keystoneauth python clients? Or do
you prefer implementing it with http requests.

And since this test requires a lot of environment set up including: 2
separate cloud installations, shibboleth, creating mapping and protocols on
remote cloud, etc. Would it be within the scope of tempest's mission?

Thank you!

Regards,
Minying Lu
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev