Back in December we had a good community-wide meeting discussing improving unit testing coverage across Fineract. We had a productive discussion on the approach to take and identified next steps that Aleks has begun to execute on as evident from the other emails he's been sending to the list.
Raw notes can be found on the wiki at: https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=195730039 I wanted to outline key decisions and next steps around implementing cucumber as test framework and leveraging community efforts to identify test cases to be created and begin implementing those test cases. We should also align this with previous discussion on creation of postman collections for test cases Michael and others have led at https://markmail.org/thread/unn6rnwypqu5fgpi and improving integration tests at https://issues.apache.org/jira/browse/FINERACT-713, https://issues.apache.org/jira/browse/FINERACT-985, https://issues.apache.org/jira/browse/FINERACT-949 Key decisions that were made: *Selection of Test Framework* - Cucumber was chosen as it enables business-driven test design and behavior-driven development where semi and non-technical individuals with more domain knowledge can easily write test cases. - Cucumber is good because you can use keywords and prose for test case definition and someone else can simply go in and fill in gaps with conditions to run test cases - Good deal of familiarity with it across members of the community. - Synchronizes well with JIRA through plug-in like https://marketplace.atlassian.com/apps/1221264/cucumber-for-jira-bdd-natively-in-jira?tab=overview&hosting=cloud *Implementation of Cucumber Framework* - Aleks to implement changes he made to build scripts to improve developer experience into upstream - Cucumber framework to then be implemented (Aleks) - In future can implement test containers for integration tests. - Ongoing changes that Petri and Aleks are working on with the REST API layer can happen in parallel *Creation of Test Cases* - Identify and separate out failing integration tests that should be in unit test cases. (Aleks) - Convert a few existing integration tests to unit tests as examples - Define the expected code coverage we'd want at a functional module leve (i.e. module should have at least 80% unit testing coverage) - Let's think about how we improve our existing integration tests (see https://issues.apache.org/jira/browse/FINERACT-949) and what's balance we want between integration tests and unit tests. - Identify which unit test cases to focus on and ultimately have a wiki page with five categories, each category having 100 to 200 test cases and red or green tick mark to show if we have coverage. - Near-term step to having this page is for individuals like Nayan and Istvan based on the test cases their teams have implemented to input into a Google Sheet at least the categories and ideally the test cases that were implemented so these can be focused on. - We can use this Google Sheet as a starting point: https://docs.google.com/spreadsheets/d/1aGTcPV_bBhyri-pEFIoKph9NKFgu4cErhCb4tvWU-rY/edit?usp=sharing - Create a test data set (No owner yet) - Create a replicable cookbook for creation of test cases using DBUnit for setting up the database and providing a few examples for others to follow. Test cases can be set up with simple CSV files. - In future can generate Jacoco and other reports for test coverage. - We can reference some previous test cases that were documented at https://drive.google.com/drive/folders/1T5MlVh0KWC6bQmqiZwEmoI73Q-0cuaxr?usp=sharing *Documentation* In parallel we need to: - Update documentation on testing - Update our strategy and guidance on what tests (unit and/or integration) are required for PR to be contributed. - It is still outstanding if test should be part of the PR or separate - Don't want to add too much friction to developers by requiring too many test to be created - Could put skeleton of test in place in Cucumber, mark as ignore test and then set up Github Actions to create a todo list of ignored tests that other volunteers could work on.
