Thanks Weiwei for the comments. Of course we should have some verification results in cases, my initial thought is to execute cases separately but your comments made me realized that maybe it's better to manage them together and provide functions as following: (1) We can keep loosely coupled relationships between framework and cases, for example, a new case can be registered into the framework and with specified type (such as benchmark, smoke, or chaos-monkey) and tags (like 'constraints', 'performance', 'fairness', 'function' ...), and provide an common entrance with which users can decide which cases to be included: all, specified type or tags, or specified one. (2) Each cases can return verification results (such as "All requirements of app1 are satisfied":succeed) to the framework and output useful information (which may helps to show execution process or locate the cause of a failure) into log file. Chaos-monkey tests can be managed in one case with type 'chaos-monkey' which has its own structure and can reuse tools in the framework. (3) After all desired test cases are done, a testing report with details of these cases can be generated on demand, an example can be: ``` Case: allocation Tags: function Status: Failed Verifications: Allocate pods with node constraint: Succeed Allocate pods with affinity constraint: Succeed Allocate pods with anti-affinity constraint: Failed ```
Above is my rough idea for that, hope to hear your thoughts. Thanks, Tao Weiwei Yang <[email protected]> 于2020年4月18日周六 上午5:44写道： > Hi Tao > > This is great, we are super interested. > Let's track this effort with > https://issues.apache.org/jira/browse/YUNIKORN-33. > A couple of comments > > 1. There might be one thing missing is, how we can make this > sustainable as "tests". which means we can 1) run this easily, 2) > generate > a result, and then 3) verify the result. Do we have #3 now? If not, we > need > to think about how to design a proper algorithm to verify if each > scenario > passes. > 2. Another thing to add is the chaos monkey tests. This can be phase-2 > but it might be good we can consider this now, to make sure our > structure > can support that. > > We can set up some meetings next week to discuss this in more detail. > Thanks > > On Fri, Apr 17, 2020 at 12:49 AM Tao Yang <[email protected]> wrote: > > > Hello everyone > > > > We are planning to add integration testing framework and initial test > cases > > in https://issues.apache.org/jira/browse/YUNIKORN-29, general thoughts > are > > as follows. > > > > Basic testing framework includes: > > 1. AppInfo struct: define basic information, requests and status of an > > application > > 2. AppManager interface and its implementations like > DeploymentAppManager: > > support useful operations like create/delete/refresh(status)/wait(for > apps > > to be satisfied or cleaned up) for testing. > > 3. CommonConfig struct: keep several common configurations for testing > like > > schedulerName, outputRootPath, namespace, queue etc. > > 4. Output tools like chart painter and file writer with specific format > > (like csv). > > > > Initial test cases: > > 1. Throughput : Request a certain number of pods via different schedulers > > and then record the distributions of scheduled pods, draw results of > > different schedulers on the same chart or write results into a file. > > 2. Node Fairness: Request a certain number of pods via different > schedulers > > and then record the distributions of node usage, draw results on charts > > separately for different schedulers and write results into a file. > > 3. Queue Fairness (Only for YuniKorn since K8s scheduler doesn't support > it > > yet): Prepare queues with different capacities, request pods with > different > > number or resource for these queues, then record the usage of queues, > draw > > results on the same chart and write results into a file. > > > > Implementation: > > 1. Package structure: add test/integration package in yunikorn-k8shim > > project, the source files with structures or tools of testing framework > > will be directly putted in this package, and test cases will be > seperately > > managed by case package in sub-directories: cases/<special-case> such as > > cases/throughput, cases/node-fairness, cases/queue-fairness > > 2. Configurations: cases keep their specific configurations themselves > and > > all configurations can be maintained in a single yaml file together with > > common configurations, the config structure like this: > > ``` > > common: > > schedulerName: yunikorn > > maxWaitSeconds: 60 > > queue: root.default > > namespace: default > > outputRootPath: /tmp > > > > cases: > > throughput: > > schedulerNames: > > - yunikorn > > - default-scheduelr > > appName: throughput-test > > numPods: 10000 > > ... > > node-fairness: > > ... > > queue-fairness: > > ... > > ``` > > 3. Executions: all cases have main function themselves and need one > > command-line argument: configPath, so that we can directly run specified > > case and easily perform integration testing on different scenarios. > > > > Any comments and suggestions are welcome! > > > > Thanks, > > Tao > > >
