Hi Markus,
I am yet to get the permission to add the proposal to the OW Wiki but though of
sharing it here:
What is Karate?
Karate is based on Cucumber which is a BDD framework.
What is BDD Testing?
BDD Testing is a testing approach based on Behavioural Driven Development. It
uses Given ,When ,Then statements to create test scenario's. Example:
TEST CASE-1
Feature: Get List of actions based on the NameSpace
Scenario: As a user I want to get the list of actions available for the given
namespace
* def path = '/api/v1/namespaces/'+nameSpace+'/actions?limit=30&skip=0'
Given url BaseUrl+path
And header Authorization = Auth
And header Content-Type = 'application/json'
When method get
Then status 200
And def json = response
The above example tests that the List Actions API's is returning a 200 ok . A lot more assertions can be put as per the requirements. Karate has some very good in built functions for
asserting the nested JSON's
TEST CASE-2(This example shows a simple smoke test on all the wsk user functions)
Feature: This feature file will test all the wsk functions
Background:
* configure ssl = true
* def nameSpace = 'guest'
* def params = '?blocking=true&result=false'
* def scriptcode = call
read('classpath:com/karate/openwhisk/functions/hello-world.js')
* def base64encoding =
read('classpath:com/karate/openwhisk/utils/base64.js')
Scenario: TC01-As a user I want to all the wsk functions available to the
user and check if they give the proper response
# Get User Auth
* def getNSCreds = call
read('classpath:com/karate/openwhisk/wskadmin/get-user.feature')
{nameSpace:'#(nameSpace)'}
* def result = getNSCreds.result
* def Auth = base64encoding(result)
* print "Got the Creds for the guest user"
* print Auth
# Create an Action .Create an action for the above defined guest name
#* def createAction = call
read('classpath:com/karate/openwhisk/wskactions/create-action.feature')
{script:'#(scriptcode)' ,nameSpace:'#(nameSpace)' ,Auth:'#(Auth)',
actionName:'Dammyyy'}
* def createAction = call
read('classpath:com/karate/openwhisk/wskactions/create-action.feature')
{script:'#(scriptcode)' ,nameSpace:'#(nameSpace)' ,Auth:'#(Auth)'}
* def actionName = createAction.actName
* print actionName
* print "Successfully Created an action"
# Get Action Details
* def actionDetails = call
read('classpath:com/karate/openwhisk/wskactions/get-action.feature')
{nameSpace:'#(nameSpace)' ,Auth:'#(Auth)',actionName:'#(actionName)'}
* print "Successfully got the action details"
#Invoke Action
* def invokeAction = call
read('classpath:com/karate/openwhisk/wskactions/invoke-action.feature')
{params:'#(params)',requestBody:'',nameSpace:'#(nameSpace)'
,Auth:'#(Auth)',actionName:'#(actionName)'}
* def actID = invokeAction.activationId
* print = "Successfully invoked the action"
* def webhooks = callonce
read('classpath:com/karate/openwhisk/utils/sleep.feature') {sheepCount:'20'}
#Get Activation details
* def getActivationDetails = call
read('classpath:com/karate/openwhisk/wskactions/get-activation-details.feature')
{ activationId: '#(actID)' ,Auth:'#(Auth)'}
* print "Successfully pulled the activation details"
# Update Action
* def updateAction = call
read('classpath:com/karate/openwhisk/wskactions/update-action.feature')
{actionName:'#(actionName)',script:'#(scriptcode)' ,nameSpace:'#(nameSpace)'
,Auth:'#(Auth)'}
* def actionName = createAction.actName
* print actionName
* print "Successfully updated the action"
# List Action
* def listActions = call
read('classpath:com/karate/openwhisk/wskactions/list-action.feature')
{nameSpace:'#(nameSpace)' ,Auth:'#(Auth)'}
* print "Successfully pulled up the list of actions"
# Delete Action
* def deleteAction = call
read('classpath:com/karate/openwhisk/wskactions/delete-action.feature')
{actionName:'#(actionName)' ,nameSpace:'#(nameSpace)' ,Auth:'#(Auth)'}
* print "Successfully deleted the action"
Some key features of BDD testing approach:
1) Shifting from thinking in “tests” to thinking in “behavior”
2) Collaboration between Business stakeholders, Business Analysts, QA Team and
developers
3) Ubiquitous language, it is easy to describe
4) Driven by Business Value
5) Extends Test Driven Development (TDD) by utilizing natural language that non
technical stakeholders can understand
6) It enables to create test cases even before development starts and also
helps bridging the Gap between what was required and what was actually
delivered.
Why we should consider doubling our test-coverage (and thus effort)?
As OpenWhisk is an opensource Serverless Platform, it offers user a wide range
of use cases. If we implement a BDD Testing Framework it would give us an
ability to create
various test flows/scenarios which mimics the user behaviour. This might help
us uncover issues which we might not catch in Unit/System level tests.
Also as writing tests in Karate does not require in-depth coding knowledge we
can expect a lot more contributions in terms of testing. While our existing
test cases can serve as Unit/System levels tests,
the Karate Framework could focus more on testing Business Logics/Work Flows.
Also,now that Karate offers Gatling support we can utilize the test feature
files for Performance Test Scenarios.
How do these tests relate to the other system tests we have written in Scala
today?
We have already covered most of the test cases in the Karate Framework and are
in Progress to create the other test cases as well. Also we are creating
units/modules derived from the Scala based Tests
which is a one time effort as those units can be utilised in any of the test
scenarios in future.
Also keeping in mind a testers perspective we have structured/grouped the tests
in the way it can be tested. Like Smoke Tests(These would just contain the
basic tests which would ensure only that that
system is working fine, Regression Tests (Covering In-depth Testing)
Below is the relation between a test case written in Scala and Karate for your
reference .The two tests give the same result.
Test Case from
(https://github.com/apache/incubator-openwhisk/blob/master/tests/src/test/scala/system/basic/WskRestBasicTests.scala)
behavior of "Wsk REST"
it should "reject creating duplicate entity" in withAssetCleaner(wskprops) {
(wp, assetHelper) =>
val name = "testDuplicateCreate"
assetHelper.withCleaner(wsk.trigger, name) { (trigger, _) =>
trigger.create(name)
}
assetHelper.withCleaner(wsk.action, name, confirmDelete = false) { (action,
_) =>
action.create(name, defaultAction, expectedExitCode = Conflict.intValue)
}
}
Test Case from (https://github.com/apache/incubator-openwhisk/blob/9d87202a9f91d6274b1922dd125fc321d1d92a3a/tests/functional/karate_tests/src/test/java/com/karate/openwhisk/actionbasictests/action-basic-tests.feature)
Scenario: As a user i want to verify the creation of duplicate entity
#create action.
* def createAction = call
read('classpath:com/karate/openwhisk/wskactions/create-action.feature')
{script:'#(scriptcode)' ,nameSpace:'#(nameSpace)' ,Auth:'#(Auth)'}
* def actionName = createAction.actName
* match createAction.responseStatusCode == 200
* print "Successfully Created an action"
#recreate the duplicate action with the same name
* def createAction = call
read('classpath:com/karate/openwhisk/wskactions/create-action.feature')
{script:'#(scriptcode)' ,nameSpace:'#(nameSpace)' ,Auth:'#(Auth)'
,actionName:'#(actionName)'}
* def actionName = createAction.actName
* match createAction.responseStatusCode == 409
* match createAction.response.error == "resource already exists"
Also if needed we will be happy to give a demo in the OW Bi-Weekly meeting.
Thanks,
Rahul
On 13/08/18, 12:34 PM, "Markus Thömmes" <[email protected]> wrote:
Hi Rahul,
thanks a lot for the contribution and the work you've put into this!
Very much appreciated.
I'm not familiar with Karate or BDD at all. Could you go into a little
bit more detail on what the benefit of this approach are and why we
should consider doubling our test-coverage (and thus effort)? How do
these tests relate to the other system tests we have written in Scala
today?
Cheers,
Markus
Am Mo., 13. Aug. 2018 um 07:55 Uhr schrieb Rahul Tripathi
<[email protected]>:
>
> Hi All,
>
> To completed the existing OpenWhisk Test I have raised this pull request
which has Karate Based BDD test cases along with Gatling Support for performance
testing. The idea is to enable the testers to contribute the user workflows in the
form of BDD scenarios.
> Request you to kindly review and provide your valuable feedback.
>
>
>
https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fincubator-openwhisk%2Fpull%2F3956&data=02%7C01%7Crtripath%40adobe.com%7C155132631fa64e50d66b08d600eaf157%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C636697406398620244&sdata=%2BdrXbT9yi6IHPcUr4eCfWicKiLfLtDsCecxxGqpLimw%3D&reserved=0
>
> Thanks,
> Rahul
>
>
>
>
> Thanks,
> Rahul
>
>
