It makes sense to make e2e driver and maybe validators in the INFRA repo. Sheng Wu 吴晟 Twitter, wusheng1108
kezhenxu94@apache <[email protected]> 于2020年12月26日周六 上午8:53写道： > Because now we have skywalking-eyes repository to host the infra tools, we > can put the new E2E framework there so that it doesn’t mess up the CLI > repo. Here [1] I created a new directory to host the E2E codes. > > [1] https://github.com/apache/skywalking-eyes/pull/11 > > ————————— > Zhenxu Ke (柯振旭) > GitHub @kezhenxu94 > > > On Dec 17, 2020, at 17:45, kezhenxu94@apache <[email protected]> > wrote: > > > > Hey, I’ve just drafted a design doc here [1], anyone interested in this > please feel free to leave your comments, thanks! > > > > [1] https://github.com/apache/skywalking-website/pull/172 > > > >> On Nov 26, 2020, at 14:41, Sheng Wu <[email protected]> wrote: > >> > >> You could learn from the Satellite project design, > >> 1. Discuss here, https://github.com/apache/skywalking/issues/5871 > >> 2. Final design posted as a blog, > >> > http://skywalking.apache.org/blog/2020-11-25-skywalking-satellite-0.1.0-design/ > >> > >> People could learn from these things when you join the community in the > >> future. > >> > >> Sheng Wu 吴晟 > >> Twitter, wusheng1108 > >> > >> > >> Hoshea Jiang <[email protected]> 于2020年11月26日周四 下午1:38写道： > >> > >>> I agree with the conclusion, and I hope to have some diagrams such as > >>> flowcharts or data flow diagrams, which help provide an overview of > the E2E > >>> test before we start to do this. > >>> > >>> Ke Zhang <[email protected]> 于2020年11月26日周四 下午12:34写道： > >>> > >>>> I think supporting both docker-compose and KIND in the E2E test is > >>>> reasonable and necessary, and I also agree with other conclusions.😀 > >>>> > >>>> kezhenxu94@apache <[email protected]> 于2020年11月26日周四 上午11:58写道： > >>>> > >>>>> > >>>>>> 1. Docker-compose and kind are 2 options to set up the environments. > >>>>>> 2. Totally replace the Java and Maven based driving system. > >>>>>> 3. Enhance CLI to validate the GraphQL > >>>>>> 4. Keep the agent test tool unchanged for now as it is already a > >>>> separate > >>>>>> system from the e2e. > >>>>> > >>>>> This conclusion is good for me. > >>>>> > >>>>> What are others opinions? > >>>>> > >>>>> ————————— > >>>>> Zhenxu Ke (柯振旭) > >>>>> GitHub @kezhenxu94 > >>>>> > >>>>>> On Nov 17, 2020, at 2:42 PM, Sheng Wu <[email protected]> > >>>> wrote: > >>>>>> > >>>>>> Using k8s and docker-compose as 2 options in the test process is > >>>>>> reasonable, and should be supported. > >>>>>> > >>>>>> Let's keep the conclusion as simple as possible. In the next > >>> generation > >>>>> e2e > >>>>>> framework > >>>>>> 1. Docker-compose and kind are 2 options to set up the environments. > >>>>>> 2. Totally replace the Java and Maven based driving system. > >>>>>> 3. Enhance CLI to validate the GraphQL > >>>>>> 4. Keep the agent test tool unchanged for now as it is already a > >>>> separate > >>>>>> system from the e2e. > >>>>>> > >>>>>> Are these good enough for everyone? > >>>>>> > >>>>>> Sheng Wu 吴晟 > >>>>>> Twitter, wusheng1108 > >>>>>> > >>>>>> > >>>>>> Hongtao Gao <[email protected]> 于2020年11月17日周二 下午12:41写道： > >>>>>> > >>>>>>>> > >>>>>>>> We can see from above, your discussion actually is only related to > >>> 3. > >>>>>>> > >>>>>>> > >>>>>>> That's the only I intend to discuss here if you notice my first > >>>>>>> response('May SWCK help about provision and demo") in this thread. > >>> For > >>>>>>> other parts, I don't have any tips to share, though. > >>>>>>> Let me explain it more clearly. I want kubernetes to be seen as an > >>>>>>> alternative to docker-compose to play a running-engine role in the > >>>> test > >>>>>>> framework. If kubernetes can't replace > >>>>>>> the latter one Is it able to participate in it to solve the > >>> dedicated > >>>> or > >>>>>>> special issues. > >>>>>>> > >>>>>>> Our e2e test and plugin test frameworks are already top-level > >>>> complicity > >>>>>>>> thing in the worldwide open source field. Please don't over-design > >>>> it, > >>>>>>> and > >>>>>>>> too aggressive, the world will stay in the hybrid env for a very > >>> long > >>>>>>> time, > >>>>>>>> same as the developer. > >>>>>>> > >>>>>>> > >>>>>>> Reusing current elaborate output might be reasonable. But in this > >>>>> thread, > >>>>>>> we're talking about the potential solution to build a > >>>>>>> next-generation framework, which might mean that the current > >>> framework > >>>>>>> is too complicated to maintain; we need a more tiny way to support > >>>> more > >>>>> and > >>>>>>> more cases. > >>>>>>> > >>>>>>> And finally, SWCK is gonna and has to build a similar test > >>> framework. > >>>> If > >>>>>>> the test framework opts for the kubernetes solution, we could share > >>>> test > >>>>>>> cases and the underlying framework. It's a more efficient > >>>>>>> and consistent path for the entire system. > >>>>>>> > >>>>>>> regards, Hongtao. > >>>>>>> > >>>>>>> Sheng Wu <[email protected]> 于2020年11月17日周二 上午10:33写道： > >>>>>>> > >>>>>>>> Hi > >>>>>>>> > >>>>>>>> I don't think we have to make the decision on k8s or out-of-k8s. > >>>>>>>> e2e and all integration tests have their scenarios like SkyWalking > >>>> have > >>>>>>>> multiple languages based and mesh-based, even Prometheus adoption. > >>>>>>>> The same thing happens on the test field, k8s, and out of k8s. > >>>>>>>> > >>>>>>>> I would like to agree with both of you, k8s is needed, no matter > in > >>>>> mesh > >>>>>>>> solution(ALS, telemetry v2), but also not anyone needs that, such > >>> as > >>>>>>> agent > >>>>>>>> developer. We wouldn't deny the fact, the docker-compose is more > >>>>>>>> lightweight and easier to learn. > >>>>>>>> So, let me get this straight, for the test you need > >>>>>>>> 1. Build the source from different repos > >>>>>>>> 2. Build the images > >>>>>>>> 3. Use some platform(docker-compose or k8s) to run the images in > >>> one > >>>>>>> piece. > >>>>>>>> 4. Give some inputs to the env > >>>>>>>> 5. Use some tools/ways to check what should happen according to > >>> (4)'s > >>>>>>> input > >>>>>>>> 6. Checked or timeout failure. > >>>>>>>> > >>>>>>>> We can see from above, your discussion actually is only related to > >>> 3. > >>>>>>> Let's > >>>>>>>> not see this as a battle, we need both. Many developers/users use > >>> OAP > >>>>> and > >>>>>>>> SkyWalking out of k8s, that is a simple fact. > >>>>>>>> Our e2e test and plugin test frameworks are already top-level > >>>>> complicity > >>>>>>>> thing in the worldwide open source field. Please don't over-design > >>>> it, > >>>>>>> and > >>>>>>>> too aggressive, the world will stay in the hybrid env for a very > >>> long > >>>>>>> time, > >>>>>>>> same as the developer. > >>>>>>>> > >>>>>>>> Also, let's keep in mind, k8s is popular becomes it doesn't > require > >>>> the > >>>>>>>> developers to understand as the VM did. We as an open-source > >>>> community, > >>>>>>> are > >>>>>>>> focusing on the developer's capabilities. > >>>>>>>> > >>>>>>>> Sheng Wu 吴晟 > >>>>>>>> Twitter, wusheng1108 > >>>>>>>> > >>>>>>>> > >>>>>>>> Hongtao Gao <[email protected]> 于2020年11月17日周二 上午12:07写道： > >>>>>>>> > >>>>>>>>>> > >>>>>>>>>> but I have been thinking whether or not it is overkill for > >>> testing > >>>>>>>>>> purposes to introduce Kubernetes things. > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> we all know the fact docker-compose is more portable than > >>>> Kubernetes, > >>>>>>>>> friendly to local running. But there're also some benefits to > >>>> replace > >>>>>>> it > >>>>>>>>> with kubernetes system: > >>>>>>>>> > >>>>>>>>> 1. SWCK has to test based on a real Kubernetes environment if the > >>>> main > >>>>>>>>> repo test framework doesn't support it(following the > >>> docker-compose > >>>>>>>> stack), > >>>>>>>>> That means skywalking ecosystem MUST introduce kubernetes which > is > >>>> not > >>>>>>> an > >>>>>>>>> optional component. > >>>>>>>>> > >>>>>>>>> 2. Kubernetes support to scale applications at runtime, that help > >>>>>>>> improve > >>>>>>>>> the process of e2e. For instance, we could merge a single > instance > >>>> and > >>>>>>>>> cluster test, by scaling up replicas. This strategy will reduce > >>> the > >>>>>>> total > >>>>>>>>> time sharply. > >>>>>>>>> > >>>>>>>>> If we are going to use KIND, I hope we can give a script or > >>>> something > >>>>>>>>>> similar to install all the needed components and create cluster > >>> for > >>>>>>>>>> convenient testing locally, Ke Zhang and Huaxi. > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> That's SWCK's capacity to provide a standard and simple way to > >>>> create > >>>>> a > >>>>>>>>> cluster in every environment, > >>>>>>>>> > >>>>>>>>> kezhenxu94@apache <[email protected]> 于2020年11月16日周一 > >>> 下午7:00写道： > >>>>>>>>> > >>>>>>>>>> Thanks Hongtao’s suggestion, I’ve thought about adopting > >>>>>>>> Kubernetes-based > >>>>>>>>>> tech stack (e.g. minikube or KIND), I personally prefer KIND > too, > >>>>>>> but I > >>>>>>>>>> have been thinking whether or not it is overkill for testing > >>>> purposes > >>>>>>>> to > >>>>>>>>>> introduce Kubernetes things. > >>>>>>>>>> > >>>>>>>>>> Since you mentioned this, we are open to discuss this much more > >>>>>>> deeply: > >>>>>>>>>> > >>>>>>>>>> It would be definitely a benefit to involve more “end”s in the > >>>>>>> so-call > >>>>>>>>>> “end to end” tests, I’m +1 to leverage SWCK as well. > >>>>>>>>>> > >>>>>>>>>> If we are going to use KIND, I hope we can give a script or > >>>> something > >>>>>>>>>> similar to install all the needed components and create cluster > >>> for > >>>>>>>>>> convenient testing locally, Ke Zhang and Huaxi. > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> ————————— > >>>>>>>>>> Zhenxu Ke (柯振旭) > >>>>>>>>>> GitHub @kezhenxu94 > >>>>>>>>>> > >>>>>>>>>>> On Nov 16, 2020, at 4:03 PM, Hongtao Gao <[email protected] > > > >>>>>>>> wrote: > >>>>>>>>>>> > >>>>>>>>>>> May SWCK help about provision and demo > >>>>>>>>>>> > >>>>>>>>>>> 1. SWCK is able to provide a stable and healthy OAP standalone > >>>>>>>> instance > >>>>>>>>>> or > >>>>>>>>>>> cluster for e2e. We could compose indicated CR yamls for > >>> different > >>>>>>>>>>> scenarios. > >>>>>>>>>>> 2. Different demo for different cases is mandatory for e2e and > >>>>>>> swck, > >>>>>>>> we > >>>>>>>>>>> could build a group of demo projects for them. With > Kubernetes's > >>>>>>>>> support, > >>>>>>>>>>> we could > >>>>>>>>>>> get free of verbose scripts that are written for installation, > >>>>>>>>>> checking > >>>>>>>>>>> the state of running applications, collecting logs. > >>>>>>>>>>> > >>>>>>>>>>> Ppl might have concerts about the minkube we currently have, > >>>>>>> which's > >>>>>>>>> hard > >>>>>>>>>>> to debug and runs slowly. > >>>>>>>>>>> > >>>>>>>>>>> I prefer to kind[1] instead of minkube. From my experience, > kind > >>>>>>>> works > >>>>>>>>>> fine > >>>>>>>>>>> in an e2e scenario(limited cpu, memory resources), and it is > run > >>>>>>>>> million > >>>>>>>>>>> times > >>>>>>>>>>> on my team's e2e environment, which proves the stability and > >>>>>>>>>>> maintainability of it. > >>>>>>>>>>> > >>>>>>>>>>> * 1.https://kind.sigs.k8s.io/docs/user/quick-start/ > >>>>>>>>>>> > >>>>>>>>>>> regards, Hongtao > >>>>>>>>>>> > >>>>>>>>>>> Sheng Wu <[email protected]> 于2020年11月16日周一 上午8:30写道： > >>>>>>>>>>> > >>>>>>>>>>>> Inline > >>>>>>>>>>>> > >>>>>>>>>>>> kezhenxu94@apache <[email protected]> 于2020年11月15日周日 > >>>>>>> 下午11:16写道： > >>>>>>>>>>>> > >>>>>>>>>>>>> Hi Jiajing, > >>>>>>>>>>>>> > >>>>>>>>>>>>> Thanks for sharing your context and joining the discussion. > >>> It’s > >>>>>>>> true > >>>>>>>>>>>> that > >>>>>>>>>>>>> we’re actually tackling the same problems in this domain. > Some > >>>> of > >>>>>>>> the > >>>>>>>>>>>>> issues you mentioned are solved in the current E2E framework > >>>>>>> (like > >>>>>>>>>>>> `retry`, > >>>>>>>>>>>>> `timeout`, etc.), while some others are still there without > >>>> ideal > >>>>>>>>>>>> solution. > >>>>>>>>>>>>> Other discussions, please see my comments inline. > >>>>>>>>>>>>> > >>>>>>>>>>>>>> - the first issue is useful in health check steps, just like > >>>> the > >>>>>>>>> ones > >>>>>>>>>>>>>> defined in readiness probe and liveness probe in kubelet. > >>>>>>>>>>>>> > >>>>>>>>>>>>> Yes, the reason why we decided to pick docker-compose is > that, > >>>> it > >>>>>>>> has > >>>>>>>>>> the > >>>>>>>>>>>>> same functionality with kubelet in terms of dependent > services > >>>>>>>>> startup > >>>>>>>>>>>>> order and healthiness checking, while it’s more lite-weight > >>> and > >>>>>>>> easy > >>>>>>>>>> for > >>>>>>>>>>>>> both developers and GitHub Actions environment. Right? > >>>>>>>>>>>>> > >>>>>>>>>>>>>> - the second issue is critical for highly-customized > >>>> assertions, > >>>>>>>> we > >>>>>>>>>>>>>> can leverage the `text/template` module to provide highly > >>>>>>>> extensive > >>>>>>>>>>>>>> assertion functions > >>>>>>>>>>>>> > >>>>>>>>>>>>> `text/template` is also the first thought that came to my > mind > >>>>>>> when > >>>>>>>>>>>>> considering customized assertors, customized assertors are > for > >>>>>>> sure > >>>>>>>>>>>>> critical in SkyWalking E2E tests because we have many kinds > of > >>>>>>>>>>>> assertions, > >>>>>>>>>>>>> some of which are not foreseen until we actually need them, > >>>>>>>>>>>>> `AtLeastOneGreaterThanZero` is an example. > >>>>>>>>>>>>> > >>>>>>>>>>>>>> - the last but not the least, the lifecycle hooks can be > used > >>>>>>> for > >>>>>>>>>>>>>> accurate control of test setup and teardown, such as > >>>>>>>> `docker-compose > >>>>>>>>>>>>>> build, up -d, down` > >>>>>>>>>>>>> > >>>>>>>>>>>>> The lifecycle hooks are what I didn’t take into consideration > >>>>>>>>> because I > >>>>>>>>>>>>> thought it is quite natural for test framework to provide > this > >>>>>>>>>> features, > >>>>>>>>>>>>> maybe Ke Zhang and Huaxi can do some research on this part. > >>>>>>>>>>>>> > >>>>>>>>>>>>>> Together with the aforementioned ideas, we will have a > >>>>>>> go-written, > >>>>>>>>>>>>>> yaml-data driven e2e testing framework which can overcome > the > >>>>>>>>> current > >>>>>>>>>>>>>> problems that occur, > >>>>>>>>>>>>> > >>>>>>>>>>>>> I agree. > >>>>>>>>>>>>> > >>>>>>>>>>>>>> in which the most annoying issue is the fragmentation of > >>>>>>> different > >>>>>>>>>>>>>> scripts in different submodules, IMO. > >>>>>>>>>>>>> > >>>>>>>>>>>>> That’s why I list the last item in the previous email, (Nice > >>> to > >>>>>>>> have, > >>>>>>>>>>>> wrap > >>>>>>>>>>>>> them into a GitHub Action), with that GitHub Action, we can > >>>>>>> simply > >>>>>>>>>>>>> configure a workflow/job in every submodule, and the tests > are > >>>>>>>>> executed > >>>>>>>>>>>>> with the shared scripts, no copy-pasting. > >>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> Note, GitHub Action will require an Apache release every time > >>>>>>>> because > >>>>>>>>> we > >>>>>>>>>>>> are distributing the binary. > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> Sheng Wu 吴晟 > >>>>>>>>>>>> Twitter, wusheng1108 > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Regards, > >>>>>>>>>>>>>> Jiajing > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> On Sun, Nov 15, 2020 at 9:43 PM kezhenxu94@apache < > >>>>>>>>>>>> [email protected]> > >>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> So, we would provide a script set to drive this? > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Yes, a script or a GoLang-written CLI, it depends. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> My question is more about, > >>>>>>>>>>>>>>>> how to work with docker-compose > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> It’s the same as what we’re doing now in the current > version > >>>> of > >>>>>>>> E2E > >>>>>>>>>>>>> tests, now we have many `docker-compose.yaml` files, and all > >>> the > >>>>>>>>>>>>> `docker-compose.yaml` files are reusable in the new > framework, > >>>> we > >>>>>>>>> will > >>>>>>>>>>>> keep > >>>>>>>>>>>>> this mechanism as it is proved to be convenient and easy to > >>> use > >>>>>>>>>> (remember > >>>>>>>>>>>>> when we added an ElasticSearch 7.9 case, and the TiDB storage > >>>> E2E > >>>>>>>>> test, > >>>>>>>>>>>> not > >>>>>>>>>>>>> yet merged though, it’s fast/easy to add a new case). > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> ————————— > >>>>>>>>>>>>>>> Zhenxu Ke (柯振旭) > >>>>>>>>>>>>>>> GitHub @kezhenxu94 > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> On Nov 15, 2020, at 9:29 PM, Sheng Wu < > >>>>>>>> [email protected]> > >>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> kezhenxu94@apache <[email protected]> 于2020年11月15日周日 > >>>>>>>>> 下午9:16写道： > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> I want to ask, how the docker-compose lands in your > >>> mind? I > >>>>>>>>> think > >>>>>>>>>>>> CLI > >>>>>>>>>>>>>>>>> can't > >>>>>>>>>>>>>>>>>> control this part. > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> docker-compose just fits the requirements perfectly that > >>> we > >>>>>>>> need, > >>>>>>>>>> we > >>>>>>>>>>>>> have > >>>>>>>>>>>>>>>>> been using it in the current E2E and turns out it has > >>>>>>>> advantages > >>>>>>>>> in > >>>>>>>>>>>>>>>>> starting many services correctly with health check and > >>>>>>>> dependency > >>>>>>>>>>>>>>>>> declaration. (One may say Kubernetes can do the same > >>> thing, > >>>>>>> but > >>>>>>>>>> it’s > >>>>>>>>>>>>> kind > >>>>>>>>>>>>>>>>> of heavy in E2E and GitHub Actions and complex). > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> The SkyWalking-CLI won’t get involved with docker-compose > >>>>>>>> related > >>>>>>>>>>>>> things, > >>>>>>>>>>>>>>>>> it just provides query ability (the same as what it is > >>> doing > >>>>>>>> now, > >>>>>>>>>>>>> just need > >>>>>>>>>>>>>>>>> to check whether it covers all the needed query > interfaces > >>>> or > >>>>>>>>> not), > >>>>>>>>>>>>> we will > >>>>>>>>>>>>>>>>> have a dedicated control program to > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> (1) start docker-compose (existing standard mechanism, no > >>>> new > >>>>>>>>>>>>> knowledge); > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> So, we would provide a script set to drive this? My > >>> question > >>>>>>> is > >>>>>>>>> more > >>>>>>>>>>>>> about, > >>>>>>>>>>>>>>>> how to work with docker-compose, rather than challanging > >>> the > >>>>>>>>> choice. > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> Sheng Wu 吴晟 > >>>>>>>>>>>>>>>> Twitter, wusheng1108 > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> (2) query actual data (by SkyWalking-CLI, existing > >>>> repository > >>>>>>>> and > >>>>>>>>>>>>> codes); > >>>>>>>>>>>>>>>>> (3) verify the actual data (by verification tool, a new > >>>> tool, > >>>>>>>>>> TODO); > >>>>>>>>>>>>>>>>> (4) and determine the test result (success or failed); > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> ————————— > >>>>>>>>>>>>>>>>> Zhenxu Ke (柯振旭) > >>>>>>>>>>>>>>>>> GitHub @kezhenxu94 > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> Sheng Wu 吴晟 > >>>>>>>>>>>>>>>>>> Twitter, wusheng1108 > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> kezhenxu94@apache <[email protected]> > 于2020年11月15日周日 > >>>>>>>>>> 下午5:27写道： > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> Hi the community, I want to raise a discussion to build > >>>> the > >>>>>>>>> next > >>>>>>>>>>>>>>>>>>> generation of E2E test framework for Apache SkyWalking, > >>>>>>> which > >>>>>>>>> may > >>>>>>>>>>>>> not > >>>>>>>>>>>>>>>>> be a > >>>>>>>>>>>>>>>>>>> high priority but a necessity. > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> As we already know, tests in SkyWalking play a very > >>>>>>> important > >>>>>>>>>> role > >>>>>>>>>>>>> in > >>>>>>>>>>>>>>>>> the > >>>>>>>>>>>>>>>>>>> contribution process, and the current E2E tests just > >>> work > >>>>>>>> well > >>>>>>>>> to > >>>>>>>>>>>>> verify > >>>>>>>>>>>>>>>>>>> every single pull request before merging, so why bother > >>> to > >>>>>>>>> build > >>>>>>>>>>>> the > >>>>>>>>>>>>>>>>> next > >>>>>>>>>>>>>>>>>>> generation of E2E test framework? > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> 1. In the SkyWalking's ecosystem, there are many > >>>>>>> sub-projects > >>>>>>>>>> that > >>>>>>>>>>>>> vary > >>>>>>>>>>>>>>>>> in > >>>>>>>>>>>>>>>>>>> terms of language, almost all of them need E2E tests to > >>>>>>> help > >>>>>>>> us > >>>>>>>>>>>>> verify > >>>>>>>>>>>>>>>>> the > >>>>>>>>>>>>>>>>>>> pull requests, but we can see that they reimplement the > >>>> E2E > >>>>>>>>> test > >>>>>>>>>>>>>>>>> framework > >>>>>>>>>>>>>>>>>>> in their languages, (or even worse, some of them don’t > >>>> have > >>>>>>>> E2E > >>>>>>>>>>>>> tests at > >>>>>>>>>>>>>>>>>>> all). Reimplementing the E2E test framework is > >>> unnecesarry > >>>>>>>> and > >>>>>>>>>>>>>>>>> introduces > >>>>>>>>>>>>>>>>>>> more duplicated work when adding a new test case. The > >>>>>>>> ultimate > >>>>>>>>>>>> goal > >>>>>>>>>>>>> is > >>>>>>>>>>>>>>>>> to > >>>>>>>>>>>>>>>>>>> reuse all the test cases when needed. > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> 2. The current E2E framework is built with Java, > running > >>>>>>>>>>>> Java-based > >>>>>>>>>>>>>>>>>>> program is not an easy thing for non-Java developers > >>>>>>>>> (maintainers > >>>>>>>>>>>> of > >>>>>>>>>>>>>>>>> other > >>>>>>>>>>>>>>>>>>> sub-projects such as PHP, C++, LUA, etc.), they need to > >>>> set > >>>>>>>> up > >>>>>>>>>> the > >>>>>>>>>>>>>>>>>>> environment correctly and then run the tests, neither > is > >>>> it > >>>>>>>> an > >>>>>>>>>>>> easy > >>>>>>>>>>>>>>>>> skill > >>>>>>>>>>>>>>>>>>> to debug the tests. > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> 3. It’s a good opportunity to optimize the E2E tests > >>>>>>> because > >>>>>>>> we > >>>>>>>>>>>>> added > >>>>>>>>>>>>>>>>> many > >>>>>>>>>>>>>>>>>>> cases and many duplicated codes that need to be > removed. > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> Therefore we're planning to build an unified, > >>> easy-to-use, > >>>>>>>> and > >>>>>>>>>>>> fast > >>>>>>>>>>>>> E2E > >>>>>>>>>>>>>>>>>>> test framework to benifit all the sub-projects. > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> Here are some rough ideas about this framework: > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> 0. (Unchanged) We will continue to use > >>> docker-compose.yaml > >>>>>>> to > >>>>>>>>>>>>>>>>> orchestrate > >>>>>>>>>>>>>>>>>>> the services that are to be tested, it’s proved to be > >>>>>>>> friendly > >>>>>>>>>> and > >>>>>>>>>>>>>>>>>>> debuggable because we can start it directly even if we > >>> are > >>>>>>>> not > >>>>>>>>>>>>> writing > >>>>>>>>>>>>>>>>> E2E > >>>>>>>>>>>>>>>>>>> tests. > >>>>>>>>>>>>>>>>>>> 1. This framework will take full advantages of the CLI > >>>>>>>> project > >>>>>>>>> to > >>>>>>>>>>>>> query > >>>>>>>>>>>>>>>>>>> the traces/metrics/logs, we don’t want to maintain two > >>>>>>>> versions > >>>>>>>>>> of > >>>>>>>>>>>>> query > >>>>>>>>>>>>>>>>>>> codes anymore, (now we have a version in the test codes > >>>> and > >>>>>>>> one > >>>>>>>>>> in > >>>>>>>>>>>>> the > >>>>>>>>>>>>>>>>> CLI). > >>>>>>>>>>>>>>>>>>> 2. We will build a CLI tool to compare the expected > data > >>>>>>> and > >>>>>>>>> the > >>>>>>>>>>>>> actual > >>>>>>>>>>>>>>>>>>> data, we now have a custom schema of the expected data > >>>> yaml > >>>>>>>>> file, > >>>>>>>>>>>>> the > >>>>>>>>>>>>>>>>>>> verification codes should be packaged into an > executable > >>>>>>>>> command > >>>>>>>>>>>> so > >>>>>>>>>>>>>>>>> that it > >>>>>>>>>>>>>>>>>>> can be executed standalone without Java and maven > >>>>>>> knowledge. > >>>>>>>> I > >>>>>>>>>>>> hope > >>>>>>>>>>>>> we > >>>>>>>>>>>>>>>>> can > >>>>>>>>>>>>>>>>>>> leverage existing standards to write the YAML files and > >>> do > >>>>>>>>>>>>>>>>> verifications. > >>>>>>>>>>>>>>>>>>> 3. Build an orchestrating tool to start the > >>>>>>>>>> `docker-compose.yaml`, > >>>>>>>>>>>>>>>>> health > >>>>>>>>>>>>>>>>>>> check, query actual data, and verify the result. > >>>>>>>>>>>>>>>>>>> 4. (Nice to have) wrap all the tools as a GitHub > Actions > >>>> to > >>>>>>>>> share > >>>>>>>>>>>>>>>>> between > >>>>>>>>>>>>>>>>>>> sub-projects. > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> If you have any other better idea or want to complement > >>> to > >>>>>>>> this > >>>>>>>>>>>>>>>>> proposal, > >>>>>>>>>>>>>>>>>>> please reply here. I will create an issue to track > these > >>>>>>>> tasks > >>>>>>>>>>>>> later. > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> We have two students (who just finished the Summer Code > >>>>>>> 2020 > >>>>>>>>>>>>> Projects) > >>>>>>>>>>>>>>>>> to > >>>>>>>>>>>>>>>>>>> lead this part, Ke Zhang [1] and Huaxi Jiang [2], as I > >>>> said > >>>>>>>>> this > >>>>>>>>>>>> is > >>>>>>>>>>>>> a > >>>>>>>>>>>>>>>>> not > >>>>>>>>>>>>>>>>>>> high priority, you have adequate time to get yourselves > >>>>>>>>> familiar > >>>>>>>>>>>>> with > >>>>>>>>>>>>>>>>> how > >>>>>>>>>>>>>>>>>>> the components (OAP, WebUI, CLI, storages, etc.) of > >>>>>>>> SkyWalking > >>>>>>>>>>>> work > >>>>>>>>>>>>> as > >>>>>>>>>>>>>>>>> well > >>>>>>>>>>>>>>>>>>> as the ecosystem. Thanks for your interests and > >>>> willingness > >>>>>>>> to > >>>>>>>>>>>> help. > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> [1] https://github.com/Humbertzhang > >>>>>>>>>>>>>>>>>>> [2] https://github.com/fgksgf/ > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> ————————— > >>>>>>>>>>>>>>>>>>> Zhenxu Ke (柯振旭) > >>>>>>>>>>>>>>>>>>> GitHub @kezhenxu94 > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> ————————— > >>>>>>>>>>>>>>>>> Zhenxu Ke (柯振旭) > >>>>>>>>>>>>>>>>> GitHub @kezhenxu94 > >>>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> ————————— > >>>>>>>>>>>>> Zhenxu Ke (柯振旭) > >>>>>>>>>>>>> GitHub @kezhenxu94 > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>>> > >>>>>>> > >>>>> > >>>>> > >>>> > >>> > > > > > > > > ————————— > > Zhenxu Ke (柯振旭) > > GitHub @kezhenxu94 > >
