Thanks Tao for initiating this thread, I am super excited to be part of this discussion.
Here are my two cents.. Framework: 1. Consider adding something like kubeconfigManager, as this allows extensibility to work with any cluster(not just with the one running locally). 2. If I have understood correctly, the nature of the framework is behavior driven. If you agree, consider something which is Ginkgo <https://onsi.github.io/ginkgo/>. It provides many inbuilt extensions and third party integrations. 3. Consider using Viper <https://github.com/spf13/viper> for configuration management, as this helps setting defaults, reading/writing config files, fileformat conversions, environment variables management etc. I have some more thoughts around test cases as well. Let us set up a meeting to discuss these points in detail, as Weiwei suggested. Thanks Ayub Khan On Sun, Apr 19, 2020 at 9:08 PM Tao Yang <[email protected]> wrote: > Thanks Weiwei for the comments. > > Of course we should have some verification results in cases, my initial > thought is to execute cases separately but your comments made me realized > that maybe it's better to manage them together and provide functions as > following: > (1) We can keep loosely coupled relationships between framework and cases, > for example, a new case can be registered into the framework and with > specified type (such as benchmark, smoke, or chaos-monkey) and tags (like > 'constraints', 'performance', 'fairness', 'function' ...), and provide an > common entrance with which users can decide which cases to be included: > all, specified type or tags, or specified one. > (2) Each cases can return verification results (such as "All requirements > of app1 are satisfied":succeed) to the framework and output useful > information (which may helps to show execution process or locate the cause > of a failure) into log file. Chaos-monkey tests can be managed in one case > with type 'chaos-monkey' which has its own structure and can reuse tools in > the framework. > (3) After all desired test cases are done, a testing report with details of > these cases can be generated on demand, an example can be: > ``` > Case: allocation > Tags: function > Status: Failed > Verifications: > Allocate pods with node constraint: Succeed > Allocate pods with affinity constraint: Succeed > Allocate pods with anti-affinity constraint: Failed > ``` > > Above is my rough idea for that, hope to hear your thoughts. > > Thanks, > Tao > > Weiwei Yang <[email protected]> 于2020年4月18日周六 上午5:44写道： > > > Hi Tao > > > > This is great, we are super interested. > > Let's track this effort with > > https://issues.apache.org/jira/browse/YUNIKORN-33. > > A couple of comments > > > > 1. There might be one thing missing is, how we can make this > > sustainable as "tests". which means we can 1) run this easily, 2) > > generate > > a result, and then 3) verify the result. Do we have #3 now? If not, we > > need > > to think about how to design a proper algorithm to verify if each > > scenario > > passes. > > 2. Another thing to add is the chaos monkey tests. This can be phase-2 > > but it might be good we can consider this now, to make sure our > > structure > > can support that. > > > > We can set up some meetings next week to discuss this in more detail. > > Thanks > > > > On Fri, Apr 17, 2020 at 12:49 AM Tao Yang <[email protected]> > wrote: > > > > > Hello everyone > > > > > > We are planning to add integration testing framework and initial test > > cases > > > in https://issues.apache.org/jira/browse/YUNIKORN-29, general thoughts > > are > > > as follows. > > > > > > Basic testing framework includes: > > > 1. AppInfo struct: define basic information, requests and status of an > > > application > > > 2. AppManager interface and its implementations like > > DeploymentAppManager: > > > support useful operations like create/delete/refresh(status)/wait(for > > apps > > > to be satisfied or cleaned up) for testing. > > > 3. CommonConfig struct: keep several common configurations for testing > > like > > > schedulerName, outputRootPath, namespace, queue etc. > > > 4. Output tools like chart painter and file writer with specific format > > > (like csv). > > > > > > Initial test cases: > > > 1. Throughput : Request a certain number of pods via different > schedulers > > > and then record the distributions of scheduled pods, draw results of > > > different schedulers on the same chart or write results into a file. > > > 2. Node Fairness: Request a certain number of pods via different > > schedulers > > > and then record the distributions of node usage, draw results on charts > > > separately for different schedulers and write results into a file. > > > 3. Queue Fairness (Only for YuniKorn since K8s scheduler doesn't > support > > it > > > yet): Prepare queues with different capacities, request pods with > > different > > > number or resource for these queues, then record the usage of queues, > > draw > > > results on the same chart and write results into a file. > > > > > > Implementation: > > > 1. Package structure: add test/integration package in yunikorn-k8shim > > > project, the source files with structures or tools of testing framework > > > will be directly putted in this package, and test cases will be > > seperately > > > managed by case package in sub-directories: cases/<special-case> such > as > > > cases/throughput, cases/node-fairness, cases/queue-fairness > > > 2. Configurations: cases keep their specific configurations themselves > > and > > > all configurations can be maintained in a single yaml file together > with > > > common configurations, the config structure like this: > > > ``` > > > common: > > > schedulerName: yunikorn > > > maxWaitSeconds: 60 > > > queue: root.default > > > namespace: default > > > outputRootPath: /tmp > > > > > > cases: > > > throughput: > > > schedulerNames: > > > - yunikorn > > > - default-scheduelr > > > appName: throughput-test > > > numPods: 10000 > > > ... > > > node-fairness: > > > ... > > > queue-fairness: > > > ... > > > ``` > > > 3. Executions: all cases have main function themselves and need one > > > command-line argument: configPath, so that we can directly run > specified > > > case and easily perform integration testing on different scenarios. > > > > > > Any comments and suggestions are welcome! > > > > > > Thanks, > > > Tao > > > > > >
