Hi,
We have come up with a mechanism to include automated tests for the REST
API which is going to be shipped with APIM 1.10.0. This is a data driven
test mechanism. The purpose of this E-Mail is to communicate the method
that we used to do the required task.
The test automation process has several java files, data files (in JSON
format)
*Data file*
Data for each scenario is stored in JSON format. Here is an example.
{
"testCase": [
{
"init": {
"scope": ""
},
"data": {
"method": "",
"url": "",
"query-parameters": "",
"request-headers": {
"Authorization": "Bearer"
},
"request-payload": "",
"response-headers": "",
"response-payload": ""
},
"asserts": {
"header-asserts": {
"status-code": ""
},
"body-asserts": ""
}
}
]
}
In a test case, there are three main sections named "init", "data" and
"asserts". Init section is used to store the user scope which is needed to
register the OAuth app and generate the access token. Data section will
store the required data for the corresponding request. It includes the HTTP
method, URL, query parameters (in nested JSON element), request-headers,
request-payload and response-headers. Asserts section is there to hold the
fields that are going to be validated against the response. Values that are
in header/body or both header and body can be compared to verify the
response.
In addition to that, we are going to include another section called
"preserve-list", if we need to take some of the values from the current
response to the next request. If this section is there, then the parameter
is stored in a Map (in the code) , as it needs to be used in subsequent
request(s).
Example : If we add (POST) a new API from the first request and if are
going to fetch it at the next request (GET), then we need to store the UUID
of the created API in the second request. In those kind of scenarios, this
"preserve-list" will be used. It contains the name of the attribute to be
preserved and the location of that attribute in the response, which is
represented in JSON format. For example, the following "preserve-list" says
that, it is needed to preserve the "apiId", which can be found in "id"
element in the response.
"preserve-list":[
{
"attribute-name":"apiId",
"response-location":"id"
}
]
If there is a resource URL like /api/am/publisher/v1//apis/{apiId}, then
the {apiId} will be fetched from the preserved list and the URL for the
request is constructed accordingly. Please note that this "preserve-list"
is optional. It is *only* needed if we need to take some value for the next
request.
The next important section is the "asserts" section. This is mainly for the
verification process to evaluate wether the test has succeeded or not.
"header-asserts" section contains the elements that should be compared
against the header of the response, while "body-asserts"
*Util Java files*
We have created util classes that can be used in any test scenario. The
util classes do the following tasks.
i) Read the data file
ii) Parse the data (i.e JSON)
iii) Register an OAuth application
iv) Generate access token (using token API)
v) Put the attributes to the preserve attribute list(if needed)
vi) Send the request to the REST API
vii) Identify the type of the request and make the relevant call (GET,POST
etc..)
viii) Get the response
These util methods are used in all test scenarios. If all the test cases
for a given scenario is successful, then we evaluate the test has passed.
Eg : If the test scenario is manipulating tiers, we first add a tier, GET
the tier, update the tier and delete the tier via the REST API. If all of
them are succeeded, we assert that the tier manipulation test has passed.
*Some more facts about this model*
If we need to add a new test scenario, all we need to do is, add a java
file only for the corresponding test, add the corresponding JSON data file
and run it. If we need to add/remove a request to/from a test scenario, all
we need to do is add/remove the required JSON configuration, save it and
run the test. Util classes won't change.
We have done R&D and discussions on selection this mechanism and we have
found that this is the most flexible approach to write test cases for the
REST API. In addition to that, we think this data driven approach can be
used in similar situations in other products as well. Eventually, we hope
this model will be beneficial when trying to come up with a automation
mechanism for all WSO2 products.
Thanks.
ChaminD
On Tue, Nov 24, 2015 at 11:47 AM, Dharshana Warusavitharana <
[email protected]> wrote:
> Hi Sanjeewa,
>
> Sorry for late reply , some how i missed this.
> Please use following versions on dependencies
>
> <org.json.wso2.version>2.0.0.wso2v1</org.json.wso2.version>
> <resteasy.version>3.0.2.Final</resteasy.version>
> <javax.ws.rs.version>2.0.1</javax.ws.rs.version>
>
>
> And place the dependency with scope of compile so that won't conflict with
> other dependencies.
>
> <dependency>
> <groupId>org.jboss.resteasy</groupId>
> <artifactId>resteasy-client</artifactId>
> <scope>compile</scope>
> </dependency>
>
> Thank you,
>
> Dharshana.
>
>
>
> On Mon, Nov 23, 2015 at 12:13 PM, Sanjeewa Malalgoda <[email protected]>
> wrote:
>
>> @dharshana can you confirm rest easy version we should use in Apim 1.10x.
>> It seems http client version conflict happen in automation tests due to
>> obove import.
>>
>> Thanks
>> sanjeewa.
>>
>> sent from my phone
>> On Nov 20, 2015 2:37 PM, "Dharshana Warusavitharana" <[email protected]>
>> wrote:
>>
>>> Hi Sanjeewa,
>>>
>>> This is ok to add these dependencies but for jax-rs please refer the
>>> jax-rs wso2 version that used in the product. Else there will be confilcts.
>>>
>>> Thank you,
>>> Dharshana.
>>>
>>> On Fri, Nov 20, 2015 at 2:29 PM, Sanjeewa Malalgoda <[email protected]>
>>> wrote:
>>>
>>>> Hi Team,
>>>> While implementing test case as per sample provided by dharshanaW i
>>>> wanted add following dependencies to test back end pom.
>>>>
>>>> <dependency>
>>>> <groupId>javax.ws.rs</groupId>
>>>> <artifactId>javax.ws.rs-api</artifactId>
>>>> <version>2.0.1</version>
>>>> </dependency>
>>>> <dependency>
>>>> <groupId>org.jboss.resteasy</groupId>
>>>> <artifactId>resteasy-client</artifactId>
>>>> <version>3.0.2.Final</version>
>>>>
>>>> Is it ok if added these dependencies to back end tests module?
>>>> We have implemented generic data driven test utility class and we may
>>>> move it to automation core as its common requirement.
>>>> Its main methods input parameters as follows.
>>>>
>>>> testRestAPI(String url, String requestPayLoad, Map<String,String>
>>>> header, Map<String, String> queryParams,
>>>> String responsePayLoad)
>>>>
>>>> Thanks,
>>>> sanjeewa.
>>>>
>>>>
>>>>
>>>> On Wed, Nov 18, 2015 at 4:58 PM, Sanjeewa Malalgoda <[email protected]>
>>>> wrote:
>>>>
>>>>> As per discussion with automation test team our plan is to implement
>>>>> data driven test cases for this.
>>>>> With that we will be able to define request data and expected response
>>>>> data.
>>>>> The utility class we are going to implement will take request data and
>>>>> then validate response with provided response.
>>>>> If the API got changed in future we do not need to change test.
>>>>> We can simply change the data set as per new API.
>>>>> This approach will help us to implement test easily and manage them.
>>>>>
>>>>> Please let us know your thought on this.
>>>>>
>>>>> Thanks,
>>>>> sanjeewa.
>>>>>
>>>>>
>>>>> On Tue, Nov 3, 2015 at 5:49 PM, Krishantha Samaraweera <
>>>>> [email protected]> wrote:
>>>>>
>>>>>> Hi Sanjeewa
>>>>>>
>>>>>> On Tue, Oct 20, 2015 at 2:47 PM, Sanjeewa Malalgoda <
>>>>>> [email protected]> wrote:
>>>>>>
>>>>>>> QAA team shall we have discussion on this and come to conclusion?
>>>>>>> Please let us know possible time slots.
>>>>>>>
>>>>>>
>>>>>> +1 shall we meet this week.
>>>>>>
>>>>>> Thanks,
>>>>>> Krishantha.
>>>>>>
>>>>>>>
>>>>>>> Thanks,
>>>>>>> sanjeewa.
>>>>>>>
>>>>>>> On Mon, Oct 19, 2015 at 12:44 PM, Sanjeewa Malalgoda <
>>>>>>> [email protected]> wrote:
>>>>>>>
>>>>>>>> Here idea is easily manage the tests and web application.
>>>>>>>> Since web application purely depend on swagger definition it will
>>>>>>>> automatically generate beans, service skeletons etc.
>>>>>>>>
>>>>>>>> So in same way if we implement test cases we don't have to go and
>>>>>>>> edit test case code after we adding new parameters or something like
>>>>>>>> that.
>>>>>>>> What we can simply do is change sample data set feed into test.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Mon, Oct 19, 2015 at 12:34 PM, Joseph Fonseka <[email protected]>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Hi Nuwan
>>>>>>>>>
>>>>>>>>> Thanks for pointing that out, in that case test can be version-ed
>>>>>>>>> so auto generated tests for version 1.0.0 should work with API 1.0.1
>>>>>>>>> ...
>>>>>>>>>
>>>>>>>>> WDYT?
>>>>>>>>>
>>>>>>>>> Regards
>>>>>>>>> Jo
>>>>>>>>>
>>>>>>>>> On Mon, Oct 19, 2015 at 12:29 PM, Nuwan Dias <[email protected]>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Is it right to auto-generate the test client code? IMO one
>>>>>>>>>> objective of this should be to make sure the REST APIs don't change
>>>>>>>>>> across
>>>>>>>>>> patch releases, etc. If we auto-generate the test stubs we would
>>>>>>>>>> loose that
>>>>>>>>>> advantage.
>>>>>>>>>>
>>>>>>>>> I don't get how we loose advantage here.
>>>>>>>> If we don't want to change service API we don't have to edit
>>>>>>>> swagger json and that is it.
>>>>>>>> Then it will not change service API, neither test client or data
>>>>>>>> set feed in to test.
>>>>>>>> If we consider both integration and unit tests those will be
>>>>>>>> released anyway with product and component release.
>>>>>>>> So tests will be automatically version-ed with component/product
>>>>>>>> version.
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> sanjeewa.
>>>>>>>>
>>>>>>>>
>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>> NuwanD.
>>>>>>>>>>
>>>>>>>>>> On Mon, Oct 19, 2015 at 12:26 PM, Joseph Fonseka <[email protected]
>>>>>>>>>> > wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi Sanjeewa
>>>>>>>>>>>
>>>>>>>>>>> Saneth & I had an offline chat regarding this last week there
>>>>>>>>>>> are few things we need to consider.
>>>>>>>>>>>
>>>>>>>>>>> 1. Generating integration test for the Jax-RS functionality ex .
>>>>>>>>>>> If crud operations work, if it returns correct error messages.
>>>>>>>>>>> - We can use swagger-codegen to do this.
>>>>>>>>>>> - API Definition has all the details of the API
>>>>>>>>>>> interface what missing are the data fixtures.
>>>>>>>>>>> - There are few options with the fixtures which we can
>>>>>>>>>>> auto generate since the schema of the model is there or we can use a
>>>>>>>>>>> predefined set of json files.
>>>>>>>>>>>
>>>>>>>>>>> 2. How to get the existing integration test to utilize the new
>>>>>>>>>>> API.
>>>>>>>>>>> - We already have a lot of integration tests which uses
>>>>>>>>>>> existing store & publisher APIs to add/remove resources. As Saneth
>>>>>>>>>>> mention
>>>>>>>>>>> they have written it in a layered architecture this replacing the
>>>>>>>>>>> existing
>>>>>>>>>>> API layer with Jax-rs existing test should work.
>>>>>>>>>>> - We might have to look at this in the next phase when
>>>>>>>>>>> we plan to deprecate the existing APIs.
>>>>>>>>>>>
>>>>>>>>>>> 3. How to make it easy to write test in the future with Jax-Rs
>>>>>>>>>>> API.
>>>>>>>>>>> - Ex Creating a Jax-rs client/library to utilize by
>>>>>>>>>>> other integration test.
>>>>>>>>>>>
>>>>>>>>>>> Thanks & Regards
>>>>>>>>>>> Jo
>>>>>>>>>>>
>>>>>>>>>>> On Mon, Oct 19, 2015 at 12:17 PM, Joseph Fonseka <
>>>>>>>>>>> [email protected]> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Sorry mail got sent accidentally half written. will complete
>>>>>>>>>>>> and send shortly.
>>>>>>>>>>>>
>>>>>>>>>>>> On Mon, Oct 19, 2015 at 12:15 PM, Joseph Fonseka <
>>>>>>>>>>>> [email protected]> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi Sanjeewa
>>>>>>>>>>>>>
>>>>>>>>>>>>> Saneth & I had an offline chat regarding this last week there
>>>>>>>>>>>>> are few things we need to consider.
>>>>>>>>>>>>>
>>>>>>>>>>>>> 1. Generating integration test for the Jax-RS functionality ex
>>>>>>>>>>>>> . If crud operations work, if it returns correct error messages.
>>>>>>>>>>>>> - We can use swagger-codegen to do this.
>>>>>>>>>>>>> API Definition has all the details of the API interface what
>>>>>>>>>>>>> missing are
>>>>>>>>>>>>>
>>>>>>>>>>>>> 2. How to get the existing integration test to utilize the new
>>>>>>>>>>>>> API.
>>>>>>>>>>>>> - We already have a lot of integration tests which
>>>>>>>>>>>>> uses existing store & publisher APIs to add/remove resources. As
>>>>>>>>>>>>> Saneth
>>>>>>>>>>>>> mention they have written it in a layered architecture this
>>>>>>>>>>>>> replacing the
>>>>>>>>>>>>> existing API layer with Jax-rs existing test should work.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> 3. How to make it easy to write test in the future with Jax-Rs
>>>>>>>>>>>>> API.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Mon, Oct 19, 2015 at 11:39 AM, Sanjeewa Malalgoda <
>>>>>>>>>>>>> [email protected]> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi Team,
>>>>>>>>>>>>>> We are planning to implement complete REST API for all
>>>>>>>>>>>>>> operations available in API Manager.
>>>>>>>>>>>>>> This will be CXF based jax-rs application.
>>>>>>>>>>>>>> This application based on swagger template and service
>>>>>>>>>>>>>> skeleton automatically generate according to swagger to cxf
>>>>>>>>>>>>>> implementation
>>>>>>>>>>>>>> done by Jo.
>>>>>>>>>>>>>> We would like to know what would be the best approach to
>>>>>>>>>>>>>> implement test for this service.
>>>>>>>>>>>>>> Do we have any mechanism to generate client and test service
>>>>>>>>>>>>>> in unit test level ? I can see there are projects to generate
>>>>>>>>>>>>>> java client
>>>>>>>>>>>>>> based on swagger content.
>>>>>>>>>>>>>> Or do we need to write integration test to run this
>>>>>>>>>>>>>> application in server and perform tests?
>>>>>>>>>>>>>> Since this implementation changes rapidly it would be ideal
>>>>>>>>>>>>>> if we can have skeleton based testing approach without binding
>>>>>>>>>>>>>> to real
>>>>>>>>>>>>>> implementation(like auto generated client).
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Have we done something similar?
>>>>>>>>>>>>>> What would be the best approach?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>> sanjeewa.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> *Sanjeewa Malalgoda*
>>>>>>>>>>>>>> WSO2 Inc.
>>>>>>>>>>>>>> Mobile : +94713068779
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> <http://sanjeewamalalgoda.blogspot.com/>blog
>>>>>>>>>>>>>> :http://sanjeewamalalgoda.blogspot.com/
>>>>>>>>>>>>>> <http://sanjeewamalalgoda.blogspot.com/>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>>> Dev mailing list
>>>>>>>>>>>>>> [email protected]
>>>>>>>>>>>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>> *Joseph Fonseka*
>>>>>>>>>>>>> WSO2 Inc.; http://wso2.com
>>>>>>>>>>>>> lean.enterprise.middleware
>>>>>>>>>>>>>
>>>>>>>>>>>>> mobile: +94 772 512 430
>>>>>>>>>>>>> skype: jpfonseka
>>>>>>>>>>>>>
>>>>>>>>>>>>> * <http://lk.linkedin.com/in/rumeshbandara>*
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> *Joseph Fonseka*
>>>>>>>>>>>> WSO2 Inc.; http://wso2.com
>>>>>>>>>>>> lean.enterprise.middleware
>>>>>>>>>>>>
>>>>>>>>>>>> mobile: +94 772 512 430
>>>>>>>>>>>> skype: jpfonseka
>>>>>>>>>>>>
>>>>>>>>>>>> * <http://lk.linkedin.com/in/rumeshbandara>*
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> *Joseph Fonseka*
>>>>>>>>>>> WSO2 Inc.; http://wso2.com
>>>>>>>>>>> lean.enterprise.middleware
>>>>>>>>>>>
>>>>>>>>>>> mobile: +94 772 512 430
>>>>>>>>>>> skype: jpfonseka
>>>>>>>>>>>
>>>>>>>>>>> * <http://lk.linkedin.com/in/rumeshbandara>*
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> _______________________________________________
>>>>>>>>>>> Dev mailing list
>>>>>>>>>>> [email protected]
>>>>>>>>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Nuwan Dias
>>>>>>>>>>
>>>>>>>>>> Technical Lead - WSO2, Inc. http://wso2.com
>>>>>>>>>> email : [email protected]
>>>>>>>>>> Phone : +94 777 775 729
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> *Joseph Fonseka*
>>>>>>>>> WSO2 Inc.; http://wso2.com
>>>>>>>>> lean.enterprise.middleware
>>>>>>>>>
>>>>>>>>> mobile: +94 772 512 430
>>>>>>>>> skype: jpfonseka
>>>>>>>>>
>>>>>>>>> * <http://lk.linkedin.com/in/rumeshbandara>*
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>>
>>>>>>>> *Sanjeewa Malalgoda*
>>>>>>>> WSO2 Inc.
>>>>>>>> Mobile : +94713068779
>>>>>>>>
>>>>>>>> <http://sanjeewamalalgoda.blogspot.com/>blog
>>>>>>>> :http://sanjeewamalalgoda.blogspot.com/
>>>>>>>> <http://sanjeewamalalgoda.blogspot.com/>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>>
>>>>>>> *Sanjeewa Malalgoda*
>>>>>>> WSO2 Inc.
>>>>>>> Mobile : +94713068779
>>>>>>>
>>>>>>> <http://sanjeewamalalgoda.blogspot.com/>blog
>>>>>>> :http://sanjeewamalalgoda.blogspot.com/
>>>>>>> <http://sanjeewamalalgoda.blogspot.com/>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Dev mailing list
>>>>>>> [email protected]
>>>>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Krishantha Samaraweera
>>>>>> Senior Technical Lead - Test Automation
>>>>>> Mobile: +94 77 7759918
>>>>>> WSO2, Inc.; http://wso2.com/
>>>>>> lean . enterprise . middleware.
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>>
>>>>> *Sanjeewa Malalgoda*
>>>>> WSO2 Inc.
>>>>> Mobile : +94713068779
>>>>>
>>>>> <http://sanjeewamalalgoda.blogspot.com/>blog
>>>>> :http://sanjeewamalalgoda.blogspot.com/
>>>>> <http://sanjeewamalalgoda.blogspot.com/>
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>>
>>>> *Sanjeewa Malalgoda*
>>>> WSO2 Inc.
>>>> Mobile : +94713068779
>>>>
>>>> <http://sanjeewamalalgoda.blogspot.com/>blog
>>>> :http://sanjeewamalalgoda.blogspot.com/
>>>> <http://sanjeewamalalgoda.blogspot.com/>
>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>>
>>> Dharshana Warusavitharana
>>> Senior Software Engineer , Test Automation
>>> WSO2 Inc. http://wso2.com
>>> email : [email protected] <[email protected]>
>>> Tel : +94 11 214 5345
>>> Fax :+94 11 2145300
>>> cell : +94770342233
>>> blog : http://dharshanaw.blogspot.com
>>>
>>> lean . enterprise . middleware
>>>
>>
>
>
> --
>
> Dharshana Warusavitharana
> Senior Software Engineer , Test Automation
> WSO2 Inc. http://wso2.com
> email : [email protected] <[email protected]>
> Tel : +94 11 214 5345
> Fax :+94 11 2145300
> cell : +94770342233
> blog : http://dharshanaw.blogspot.com
>
> lean . enterprise . middleware
>
--
Chamin Dias
*Software Engineer*
Mobile : +94 (0) 716 097455 <%2B94%20%280%29%20773%20451194>
Email : [email protected]
Blog : https://chamindias.wordpress.com/
_______________________________________________
Dev mailing list
[email protected]
http://wso2.org/cgi-bin/mailman/listinfo/dev
