[openstack-dev] [qa] [Solum] [tempest] Use of pecan test framework in functional tests

2013-12-10 Thread Georgy Okrokvertskhov
Hi,

In Solum project we are currently creating tests environments for future
test. We split unit tests and functional tests in order to use tempest
framework from the beginning.

Tempest framework assumes that you run your service and test APi endpoints
by sending HTTP requests. Solum uses Pecan WSGI framework which has its own
test framework based on WebTest. This framework allows to test application
without sending actual HTTP traffic. It mocks low level stuff related to
transport but keeps all high level WSGI part as it is a real life
application\service.

There is a question to QA\Tempest teams, what do you think about using
pecan test framework in tempest for Pecan based applications?

Thanks
Georgy
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [qa] [Solum] [tempest] Use of pecan test framework in functional tests

2013-12-10 Thread Russell Bryant
On 12/10/2013 04:10 PM, Georgy Okrokvertskhov wrote:
 Hi,
 
 In Solum project we are currently creating tests environments for future
 test. We split unit tests and functional tests in order to use tempest
 framework from the beginning. 
 
 Tempest framework assumes that you run your service and test APi
 endpoints by sending HTTP requests. Solum uses Pecan WSGI framework
 which has its own test framework based on WebTest. This framework allows
 to test application without sending actual HTTP traffic. It mocks low
 level stuff related to transport but keeps all high level WSGI part as
 it is a real life application\service.
 
 There is a question to QA\Tempest teams, what do you think about using
 pecan test framework in tempest for Pecan based applications?

I don't think that makes sense.  Then we're not using the code like it
would be used normally (via HTTP).

-- 
Russell Bryant

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [qa] [Solum] [tempest] Use of pecan test framework in functional tests

2013-12-10 Thread Ryan Petrello
My opinion is that there’s value in both.  Writing functional tests for Solum’s 
test suite using WebTest can be pretty useful for testing the API’s logic 
without having to involve HTTP (to e.g., call API endpoints with certain POST 
arguments and assert that certain mocked functions end up being called down the 
line).

When you involve Tempest, though, you’re generally pointing at a real HTTP 
server and testing for correctness, so using HTTP here makes sense (imo).

---
Ryan Petrello
Senior Developer, DreamHost
ryan.petre...@dreamhost.com

On Dec 10, 2013, at 4:12 PM, Russell Bryant rbry...@redhat.com wrote:

 On 12/10/2013 04:10 PM, Georgy Okrokvertskhov wrote:
 Hi,
 
 In Solum project we are currently creating tests environments for future
 test. We split unit tests and functional tests in order to use tempest
 framework from the beginning. 
 
 Tempest framework assumes that you run your service and test APi
 endpoints by sending HTTP requests. Solum uses Pecan WSGI framework
 which has its own test framework based on WebTest. This framework allows
 to test application without sending actual HTTP traffic. It mocks low
 level stuff related to transport but keeps all high level WSGI part as
 it is a real life application\service.
 
 There is a question to QA\Tempest teams, what do you think about using
 pecan test framework in tempest for Pecan based applications?
 
 I don't think that makes sense.  Then we're not using the code like it
 would be used normally (via HTTP).
 
 -- 
 Russell Bryant
 
 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [qa] [Solum] [tempest] Use of pecan test framework in functional tests

2013-12-10 Thread David Kranz

On 12/10/2013 04:12 PM, Russell Bryant wrote:

On 12/10/2013 04:10 PM, Georgy Okrokvertskhov wrote:

Hi,

In Solum project we are currently creating tests environments for future
test. We split unit tests and functional tests in order to use tempest
framework from the beginning.

Tempest framework assumes that you run your service and test APi
endpoints by sending HTTP requests. Solum uses Pecan WSGI framework
which has its own test framework based on WebTest. This framework allows
to test application without sending actual HTTP traffic. It mocks low
level stuff related to transport but keeps all high level WSGI part as
it is a real life application\service.

There is a question to QA\Tempest teams, what do you think about using
pecan test framework in tempest for Pecan based applications?

I don't think that makes sense.  Then we're not using the code like it
would be used normally (via HTTP).

+1 There are other projects that do things like this as well in their 
unit tests. But Tempest is supposed to test the real API in a blackbox 
fashion and protect against unintentional api changes. It is also 
proposed to be used with refstack.


And to clarify, Tempest uses REST apis for the api stability tests. The 
scenario tests use the official python clients.


 -David

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [qa] [Solum] [tempest] Use of pecan test framework in functional tests

2013-12-10 Thread Sean Dague
Pretty much 100% agree with Russell and Ryan.

Webtest is interesting for in tree testing with Solum, because it's
specifically *not* bringing up the full stack.

When it comes to Tempest, you are hitting a live OpenStack cloud, most
likely not on the same machine as Tempest is on (not true in the gate
today... but we try to act like it is). So you must hit HTTP.

-Sean

On 12/10/2013 04:24 PM, Ryan Petrello wrote:
 My opinion is that there’s value in both.  Writing functional tests for 
 Solum’s test suite using WebTest can be pretty useful for testing the API’s 
 logic without having to involve HTTP (to e.g., call API endpoints with 
 certain POST arguments and assert that certain mocked functions end up being 
 called down the line).
 
 When you involve Tempest, though, you’re generally pointing at a real HTTP 
 server and testing for correctness, so using HTTP here makes sense (imo).
 
 ---
 Ryan Petrello
 Senior Developer, DreamHost
 ryan.petre...@dreamhost.com
 
 On Dec 10, 2013, at 4:12 PM, Russell Bryant rbry...@redhat.com wrote:
 
 On 12/10/2013 04:10 PM, Georgy Okrokvertskhov wrote:
 Hi,

 In Solum project we are currently creating tests environments for future
 test. We split unit tests and functional tests in order to use tempest
 framework from the beginning. 

 Tempest framework assumes that you run your service and test APi
 endpoints by sending HTTP requests. Solum uses Pecan WSGI framework
 which has its own test framework based on WebTest. This framework allows
 to test application without sending actual HTTP traffic. It mocks low
 level stuff related to transport but keeps all high level WSGI part as
 it is a real life application\service.

 There is a question to QA\Tempest teams, what do you think about using
 pecan test framework in tempest for Pecan based applications?

 I don't think that makes sense.  Then we're not using the code like it
 would be used normally (via HTTP).

 -- 
 Russell Bryant

 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
 
 
 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
 


-- 
Sean Dague
http://dague.net



signature.asc
Description: OpenPGP digital signature
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [qa] [Solum] [tempest] Use of pecan test framework in functional tests

2013-12-10 Thread Georgy Okrokvertskhov
Thanks everyone for feedback. Will follow the standard approach with HTTP
requests in tempest tests.

Thanks
Georgy


On Tue, Dec 10, 2013 at 2:47 PM, Sean Dague s...@dague.net wrote:

 Pretty much 100% agree with Russell and Ryan.

 Webtest is interesting for in tree testing with Solum, because it's
 specifically *not* bringing up the full stack.

 When it comes to Tempest, you are hitting a live OpenStack cloud, most
 likely not on the same machine as Tempest is on (not true in the gate
 today... but we try to act like it is). So you must hit HTTP.

 -Sean

 On 12/10/2013 04:24 PM, Ryan Petrello wrote:
  My opinion is that there’s value in both.  Writing functional tests for
 Solum’s test suite using WebTest can be pretty useful for testing the API’s
 logic without having to involve HTTP (to e.g., call API endpoints with
 certain POST arguments and assert that certain mocked functions end up
 being called down the line).
 
  When you involve Tempest, though, you’re generally pointing at a real
 HTTP server and testing for correctness, so using HTTP here makes sense
 (imo).
 
  ---
  Ryan Petrello
  Senior Developer, DreamHost
  ryan.petre...@dreamhost.com
 
  On Dec 10, 2013, at 4:12 PM, Russell Bryant rbry...@redhat.com wrote:
 
  On 12/10/2013 04:10 PM, Georgy Okrokvertskhov wrote:
  Hi,
 
  In Solum project we are currently creating tests environments for
 future
  test. We split unit tests and functional tests in order to use tempest
  framework from the beginning.
 
  Tempest framework assumes that you run your service and test APi
  endpoints by sending HTTP requests. Solum uses Pecan WSGI framework
  which has its own test framework based on WebTest. This framework
 allows
  to test application without sending actual HTTP traffic. It mocks low
  level stuff related to transport but keeps all high level WSGI part as
  it is a real life application\service.
 
  There is a question to QA\Tempest teams, what do you think about using
  pecan test framework in tempest for Pecan based applications?
 
  I don't think that makes sense.  Then we're not using the code like it
  would be used normally (via HTTP).
 
  --
  Russell Bryant
 
  ___
  OpenStack-dev mailing list
  OpenStack-dev@lists.openstack.org
  http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
 
 
  ___
  OpenStack-dev mailing list
  OpenStack-dev@lists.openstack.org
  http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
 


 --
 Sean Dague
 http://dague.net


 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




-- 
Georgy Okrokvertskhov
Technical Program Manager,
Cloud and Infrastructure Services,
Mirantis
http://www.mirantis.com
Tel. +1 650 963 9828
Mob. +1 650 996 3284
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [qa] [Solum] [tempest] Use of pecan test framework in functional tests

2013-12-10 Thread Adrian Otto
On Dec 10, 2013, at 3:12 PM, Russell Bryant rbry...@redhat.com
 wrote:

 On 12/10/2013 04:10 PM, Georgy Okrokvertskhov wrote:
 Hi,
 
 In Solum project we are currently creating tests environments for future
 test. We split unit tests and functional tests in order to use tempest
 framework from the beginning. 
 
 Tempest framework assumes that you run your service and test APi
 endpoints by sending HTTP requests. Solum uses Pecan WSGI framework
 which has its own test framework based on WebTest. This framework allows
 to test application without sending actual HTTP traffic. It mocks low
 level stuff related to transport but keeps all high level WSGI part as
 it is a real life application\service.
 
 There is a question to QA\Tempest teams, what do you think about using
 pecan test framework in tempest for Pecan based applications?
 
 I don't think that makes sense.  Then we're not using the code like it
 would be used normally (via HTTP).

I agree. The benefits we might get from bypassing HTTP for testing would likely 
be limited to speed of execution of the test, right? Unless we are suffering 
productivity problems from slow tests, I'd rather test it through HTTP just 
like a user would experience it.

Adrian


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev