Thanks Ayub for the suggestions, I think they are all very useful for setting up a testing framework. Looking forward to join the meeting to have a further discussion, @Weiwei Yang <[email protected]>, could you please help to organize it?
Thanks, Tao Ayub Pathan <[email protected]> 于2020年4月21日周二 上午8:28写道： > Thanks Tao for initiating this thread, I am super excited to be part of > this discussion. > > Here are my two cents.. > > Framework: > > 1. Consider adding something like kubeconfigManager, as this allows > extensibility to work with any cluster(not just with the one running > locally). > 2. If I have understood correctly, the nature of the framework is > behavior driven. If you agree, consider something which is Ginkgo > <https://onsi.github.io/ginkgo/>. It provides many > inbuilt extensions and third party integrations. > 3. Consider using Viper <https://github.com/spf13/viper> for > configuration management, as this helps setting defaults, > reading/writing > config files, fileformat conversions, environment variables management > etc. > > I have some more thoughts around test cases as well. Let us set up a > meeting to discuss these points in detail, as Weiwei suggested. > > Thanks > Ayub Khan > > On Sun, Apr 19, 2020 at 9:08 PM Tao Yang <[email protected]> wrote: > > > Thanks Weiwei for the comments. > > > > Of course we should have some verification results in cases, my initial > > thought is to execute cases separately but your comments made me realized > > that maybe it's better to manage them together and provide functions as > > following: > > (1) We can keep loosely coupled relationships between framework and > cases, > > for example, a new case can be registered into the framework and with > > specified type (such as benchmark, smoke, or chaos-monkey) and tags (like > > 'constraints', 'performance', 'fairness', 'function' ...), and provide an > > common entrance with which users can decide which cases to be included: > > all, specified type or tags, or specified one. > > (2) Each cases can return verification results (such as "All requirements > > of app1 are satisfied":succeed) to the framework and output useful > > information (which may helps to show execution process or locate the > cause > > of a failure) into log file. Chaos-monkey tests can be managed in one > case > > with type 'chaos-monkey' which has its own structure and can reuse tools > in > > the framework. > > (3) After all desired test cases are done, a testing report with details > of > > these cases can be generated on demand, an example can be: > > ``` > > Case: allocation > > Tags: function > > Status: Failed > > Verifications: > > Allocate pods with node constraint: Succeed > > Allocate pods with affinity constraint: Succeed > > Allocate pods with anti-affinity constraint: Failed > > ``` > > > > Above is my rough idea for that, hope to hear your thoughts. > > > > Thanks, > > Tao > > > > Weiwei Yang <[email protected]> 于2020年4月18日周六 上午5:44写道： > > > > > Hi Tao > > > > > > This is great, we are super interested. > > > Let's track this effort with > > > https://issues.apache.org/jira/browse/YUNIKORN-33. > > > A couple of comments > > > > > > 1. There might be one thing missing is, how we can make this > > > sustainable as "tests". which means we can 1) run this easily, 2) > > > generate > > > a result, and then 3) verify the result. Do we have #3 now? If not, > we > > > need > > > to think about how to design a proper algorithm to verify if each > > > scenario > > > passes. > > > 2. Another thing to add is the chaos monkey tests. This can be > phase-2 > > > but it might be good we can consider this now, to make sure our > > > structure > > > can support that. > > > > > > We can set up some meetings next week to discuss this in more detail. > > > Thanks > > > > > > On Fri, Apr 17, 2020 at 12:49 AM Tao Yang <[email protected]> > > wrote: > > > > > > > Hello everyone > > > > > > > > We are planning to add integration testing framework and initial test > > > cases > > > > in https://issues.apache.org/jira/browse/YUNIKORN-29, general > thoughts > > > are > > > > as follows. > > > > > > > > Basic testing framework includes: > > > > 1. AppInfo struct: define basic information, requests and status of > an > > > > application > > > > 2. AppManager interface and its implementations like > > > DeploymentAppManager: > > > > support useful operations like create/delete/refresh(status)/wait(for > > > apps > > > > to be satisfied or cleaned up) for testing. > > > > 3. CommonConfig struct: keep several common configurations for > testing > > > like > > > > schedulerName, outputRootPath, namespace, queue etc. > > > > 4. Output tools like chart painter and file writer with specific > format > > > > (like csv). > > > > > > > > Initial test cases: > > > > 1. Throughput : Request a certain number of pods via different > > schedulers > > > > and then record the distributions of scheduled pods, draw results of > > > > different schedulers on the same chart or write results into a file. > > > > 2. Node Fairness: Request a certain number of pods via different > > > schedulers > > > > and then record the distributions of node usage, draw results on > charts > > > > separately for different schedulers and write results into a file. > > > > 3. Queue Fairness (Only for YuniKorn since K8s scheduler doesn't > > support > > > it > > > > yet): Prepare queues with different capacities, request pods with > > > different > > > > number or resource for these queues, then record the usage of queues, > > > draw > > > > results on the same chart and write results into a file. > > > > > > > > Implementation: > > > > 1. Package structure: add test/integration package in yunikorn-k8shim > > > > project, the source files with structures or tools of testing > framework > > > > will be directly putted in this package, and test cases will be > > > seperately > > > > managed by case package in sub-directories: cases/<special-case> such > > as > > > > cases/throughput, cases/node-fairness, cases/queue-fairness > > > > 2. Configurations: cases keep their specific configurations > themselves > > > and > > > > all configurations can be maintained in a single yaml file together > > with > > > > common configurations, the config structure like this: > > > > ``` > > > > common: > > > > schedulerName: yunikorn > > > > maxWaitSeconds: 60 > > > > queue: root.default > > > > namespace: default > > > > outputRootPath: /tmp > > > > > > > > cases: > > > > throughput: > > > > schedulerNames: > > > > - yunikorn > > > > - default-scheduelr > > > > appName: throughput-test > > > > numPods: 10000 > > > > ... > > > > node-fairness: > > > > ... > > > > queue-fairness: > > > > ... > > > > ``` > > > > 3. Executions: all cases have main function themselves and need one > > > > command-line argument: configPath, so that we can directly run > > specified > > > > case and easily perform integration testing on different scenarios. > > > > > > > > Any comments and suggestions are welcome! > > > > > > > > Thanks, > > > > Tao > > > > > > > > > >
