I think supporting both docker-compose and KIND in the E2E test is reasonable and necessary, and I also agree with other conclusions.π
kezhenxu94@apache <[email protected]> δΊ2020εΉ΄11ζ26ζ₯ε¨ε δΈε11:58ειοΌ > > > 1. Docker-compose and kind are 2 options to set up the environments. > > 2. Totally replace the Java and Maven based driving system. > > 3. Enhance CLI to validate the GraphQL > > 4. Keep the agent test tool unchanged for now as it is already a separate > > system from the e2e. > > This conclusion is good for me. > > What are others opinions? > > βββββββββ > Zhenxu Ke (ζ―ζ―ζ) > GitHub @kezhenxu94 > > > On Nov 17, 2020, at 2:42 PM, Sheng Wu <[email protected]> wrote: > > > > Using k8s and docker-compose as 2 options in the test process is > > reasonable, and should be supported. > > > > Let's keep the conclusion as simple as possible. In the next generation > e2e > > framework > > 1. Docker-compose and kind are 2 options to set up the environments. > > 2. Totally replace the Java and Maven based driving system. > > 3. Enhance CLI to validate the GraphQL > > 4. Keep the agent test tool unchanged for now as it is already a separate > > system from the e2e. > > > > Are these good enough for everyone? > > > > Sheng Wu ε΄ζ > > Twitter, wusheng1108 > > > > > > Hongtao Gao <[email protected]> δΊ2020εΉ΄11ζ17ζ₯ε¨δΊ δΈε12:41ειοΌ > > > >>> > >>> We can see from above, your discussion actually is only related to 3. > >> > >> > >> That's the only I intend to discuss here if you notice my first > >> response('May SWCK help about provision and demo") in this thread. For > >> other parts, I don't have any tips to share, though. > >> Let me explain it more clearly. I want kubernetes to be seen as an > >> alternative to docker-compose to play a running-engine role in the test > >> framework. If kubernetes can't replace > >> the latter one Is it able to participate in it to solve the dedicated or > >> special issues. > >> > >> Our e2e test and plugin test frameworks are already top-level complicity > >>> thing in the worldwide open source field. Please don't over-design it, > >> and > >>> too aggressive, the world will stay in the hybrid env for a very long > >> time, > >>> same as the developer. > >> > >> > >> Reusing current elaborate output might be reasonable. But in this > thread, > >> we're talking about the potential solution to build a > >> next-generation framework, which might mean that the current framework > >> is too complicated to maintain; we need a more tiny way to support more > and > >> more cases. > >> > >> And finally, SWCK is gonna and has to build a similar test framework. If > >> the test framework opts for the kubernetes solution, we could share test > >> cases and the underlying framework. It's a more efficient > >> and consistent path for the entire system. > >> > >> regards, Hongtao. > >> > >> Sheng Wu <[email protected]> δΊ2020εΉ΄11ζ17ζ₯ε¨δΊ δΈε10:33ειοΌ > >> > >>> Hi > >>> > >>> I don't think we have to make the decision on k8s or out-of-k8s. > >>> e2e and all integration tests have their scenarios like SkyWalking have > >>> multiple languages based and mesh-based, even Prometheus adoption. > >>> The same thing happens on the test field, k8s, and out of k8s. > >>> > >>> I would like to agree with both of you, k8s is needed, no matter in > mesh > >>> solution(ALS, telemetry v2), but also not anyone needs that, such as > >> agent > >>> developer. We wouldn't deny the fact, the docker-compose is more > >>> lightweight and easier to learn. > >>> So, let me get this straight, for the test you need > >>> 1. Build the source from different repos > >>> 2. Build the images > >>> 3. Use some platform(docker-compose or k8s) to run the images in one > >> piece. > >>> 4. Give some inputs to the env > >>> 5. Use some tools/ways to check what should happen according to (4)'s > >> input > >>> 6. Checked or timeout failure. > >>> > >>> We can see from above, your discussion actually is only related to 3. > >> Let's > >>> not see this as a battle, we need both. Many developers/users use OAP > and > >>> SkyWalking out of k8s, that is a simple fact. > >>> Our e2e test and plugin test frameworks are already top-level > complicity > >>> thing in the worldwide open source field. Please don't over-design it, > >> and > >>> too aggressive, the world will stay in the hybrid env for a very long > >> time, > >>> same as the developer. > >>> > >>> Also, let's keep in mind, k8s is popular becomes it doesn't require the > >>> developers to understand as the VM did. We as an open-source community, > >> are > >>> focusing on the developer's capabilities. > >>> > >>> Sheng Wu ε΄ζ > >>> Twitter, wusheng1108 > >>> > >>> > >>> Hongtao Gao <[email protected]> δΊ2020εΉ΄11ζ17ζ₯ε¨δΊ δΈε12:07ειοΌ > >>> > >>>>> > >>>>> but I have been thinking whether or not it is overkill for testing > >>>>> purposes to introduce Kubernetes things. > >>>> > >>>> > >>>> we all know the fact docker-compose is more portable than Kubernetes, > >>>> friendly to local running. But there're also some benefits to replace > >> it > >>>> with kubernetes system: > >>>> > >>>> 1. SWCK has to test based on a real Kubernetes environment if the main > >>>> repo test framework doesn't support it(following the docker-compose > >>> stack), > >>>> That means skywalking ecosystem MUST introduce kubernetes which is not > >> an > >>>> optional component. > >>>> > >>>> 2. Kubernetes support to scale applications at runtime, that help > >>> improve > >>>> the process of e2e. For instance, we could merge a single instance and > >>>> cluster test, by scaling up replicas. This strategy will reduce the > >> total > >>>> time sharply. > >>>> > >>>> If we are going to use KIND, I hope we can give a script or something > >>>>> similar to install all the needed components and create cluster for > >>>>> convenient testing locally, Ke Zhang and Huaxi. > >>>> > >>>> > >>>> That's SWCK's capacity to provide a standard and simple way to create > a > >>>> cluster in every environment, > >>>> > >>>> kezhenxu94@apache <[email protected]> δΊ2020εΉ΄11ζ16ζ₯ε¨δΈ δΈε7:00ειοΌ > >>>> > >>>>> Thanks Hongtaoβs suggestion, Iβve thought about adopting > >>> Kubernetes-based > >>>>> tech stack (e.g. minikube or KIND), I personally prefer KIND too, > >> but I > >>>>> have been thinking whether or not it is overkill for testing purposes > >>> to > >>>>> introduce Kubernetes things. > >>>>> > >>>>> Since you mentioned this, we are open to discuss this much more > >> deeply: > >>>>> > >>>>> It would be definitely a benefit to involve more βendβs in the > >> so-call > >>>>> βend to endβ tests, Iβm +1 to leverage SWCK as well. > >>>>> > >>>>> If we are going to use KIND, I hope we can give a script or something > >>>>> similar to install all the needed components and create cluster for > >>>>> convenient testing locally, Ke Zhang and Huaxi. > >>>>> > >>>>> > >>>>> βββββββββ > >>>>> Zhenxu Ke (ζ―ζ―ζ) > >>>>> GitHub @kezhenxu94 > >>>>> > >>>>>> On Nov 16, 2020, at 4:03 PM, Hongtao Gao <[email protected]> > >>> wrote: > >>>>>> > >>>>>> May SWCK help about provision and demo > >>>>>> > >>>>>> 1. SWCK is able to provide a stable and healthy OAP standalone > >>> instance > >>>>> or > >>>>>> cluster for e2e. We could compose indicated CR yamls for different > >>>>>> scenarios. > >>>>>> 2. Different demo for different cases is mandatory for e2e and > >> swck, > >>> we > >>>>>> could build a group of demo projects for them. With Kubernetes's > >>>> support, > >>>>>> we could > >>>>>> get free of verbose scripts that are written for installation, > >>>>> checking > >>>>>> the state of running applications, collecting logs. > >>>>>> > >>>>>> Ppl might have concerts about the minkube we currently have, > >> which's > >>>> hard > >>>>>> to debug and runs slowly. > >>>>>> > >>>>>> I prefer to kind[1] instead of minkube. From my experience, kind > >>> works > >>>>> fine > >>>>>> in an e2e scenario(limited cpu, memory resources), and it is run > >>>> million > >>>>>> times > >>>>>> on my team's e2e environment, which proves the stability and > >>>>>> maintainability of it. > >>>>>> > >>>>>> * 1.https://kind.sigs.k8s.io/docs/user/quick-start/ > >>>>>> > >>>>>> regards, Hongtao > >>>>>> > >>>>>> Sheng Wu <[email protected]> δΊ2020εΉ΄11ζ16ζ₯ε¨δΈ δΈε8:30ειοΌ > >>>>>> > >>>>>>> Inline > >>>>>>> > >>>>>>> kezhenxu94@apache <[email protected]> δΊ2020εΉ΄11ζ15ζ₯ε¨ζ₯ > >> δΈε11:16ειοΌ > >>>>>>> > >>>>>>>> Hi Jiajing, > >>>>>>>> > >>>>>>>> Thanks for sharing your context and joining the discussion. Itβs > >>> true > >>>>>>> that > >>>>>>>> weβre actually tackling the same problems in this domain. Some of > >>> the > >>>>>>>> issues you mentioned are solved in the current E2E framework > >> (like > >>>>>>> `retry`, > >>>>>>>> `timeout`, etc.), while some others are still there without ideal > >>>>>>> solution. > >>>>>>>> Other discussions, please see my comments inline. > >>>>>>>> > >>>>>>>>> - the first issue is useful in health check steps, just like the > >>>> ones > >>>>>>>>> defined in readiness probe and liveness probe in kubelet. > >>>>>>>> > >>>>>>>> Yes, the reason why we decided to pick docker-compose is that, it > >>> has > >>>>> the > >>>>>>>> same functionality with kubelet in terms of dependent services > >>>> startup > >>>>>>>> order and healthiness checking, while itβs more lite-weight and > >>> easy > >>>>> for > >>>>>>>> both developers and GitHub Actions environment. Right? > >>>>>>>> > >>>>>>>>> - the second issue is critical for highly-customized assertions, > >>> we > >>>>>>>>> can leverage the `text/template` module to provide highly > >>> extensive > >>>>>>>>> assertion functions > >>>>>>>> > >>>>>>>> `text/template` is also the first thought that came to my mind > >> when > >>>>>>>> considering customized assertors, customized assertors are for > >> sure > >>>>>>>> critical in SkyWalking E2E tests because we have many kinds of > >>>>>>> assertions, > >>>>>>>> some of which are not foreseen until we actually need them, > >>>>>>>> `AtLeastOneGreaterThanZero` is an example. > >>>>>>>> > >>>>>>>>> - the last but not the least, the lifecycle hooks can be used > >> for > >>>>>>>>> accurate control of test setup and teardown, such as > >>> `docker-compose > >>>>>>>>> build, up -d, down` > >>>>>>>> > >>>>>>>> The lifecycle hooks are what I didnβt take into consideration > >>>> because I > >>>>>>>> thought it is quite natural for test framework to provide this > >>>>> features, > >>>>>>>> maybe Ke Zhang and Huaxi can do some research on this part. > >>>>>>>> > >>>>>>>>> Together with the aforementioned ideas, we will have a > >> go-written, > >>>>>>>>> yaml-data driven e2e testing framework which can overcome the > >>>> current > >>>>>>>>> problems that occur, > >>>>>>>> > >>>>>>>> I agree. > >>>>>>>> > >>>>>>>>> in which the most annoying issue is the fragmentation of > >> different > >>>>>>>>> scripts in different submodules, IMO. > >>>>>>>> > >>>>>>>> Thatβs why I list the last item in the previous email, (Nice to > >>> have, > >>>>>>> wrap > >>>>>>>> them into a GitHub Action), with that GitHub Action, we can > >> simply > >>>>>>>> configure a workflow/job in every submodule, and the tests are > >>>> executed > >>>>>>>> with the shared scripts, no copy-pasting. > >>>>>>>> > >>>>>>> > >>>>>>> Note, GitHub Action will require an Apache release every time > >>> because > >>>> we > >>>>>>> are distributing the binary. > >>>>>>> > >>>>>>> > >>>>>>> Sheng Wu ε΄ζ > >>>>>>> Twitter, wusheng1108 > >>>>>>> > >>>>>>> > >>>>>>>> > >>>>>>>>> > >>>>>>>>> Regards, > >>>>>>>>> Jiajing > >>>>>>>>> > >>>>>>>>> On Sun, Nov 15, 2020 at 9:43 PM kezhenxu94@apache < > >>>>>>> [email protected]> > >>>>>>>> wrote: > >>>>>>>>>> > >>>>>>>>>>> So, we would provide a script set to drive this? > >>>>>>>>>> > >>>>>>>>>> Yes, a script or a GoLang-written CLI, it depends. > >>>>>>>>>> > >>>>>>>>>>> My question is more about, > >>>>>>>>>>> how to work with docker-compose > >>>>>>>>>> > >>>>>>>>>> Itβs the same as what weβre doing now in the current version of > >>> E2E > >>>>>>>> tests, now we have many `docker-compose.yaml` files, and all the > >>>>>>>> `docker-compose.yaml` files are reusable in the new framework, we > >>>> will > >>>>>>> keep > >>>>>>>> this mechanism as it is proved to be convenient and easy to use > >>>>> (remember > >>>>>>>> when we added an ElasticSearch 7.9 case, and the TiDB storage E2E > >>>> test, > >>>>>>> not > >>>>>>>> yet merged though, itβs fast/easy to add a new case). > >>>>>>>>>> > >>>>>>>>>> βββββββββ > >>>>>>>>>> Zhenxu Ke (ζ―ζ―ζ) > >>>>>>>>>> GitHub @kezhenxu94 > >>>>>>>>>> > >>>>>>>>>>> On Nov 15, 2020, at 9:29 PM, Sheng Wu < > >>> [email protected]> > >>>>>>>> wrote: > >>>>>>>>>>> > >>>>>>>>>>> kezhenxu94@apache <[email protected]> δΊ2020εΉ΄11ζ15ζ₯ε¨ζ₯ > >>>> δΈε9:16ειοΌ > >>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>>> I want to ask, how the docker-compose lands in your mind? I > >>>> think > >>>>>>> CLI > >>>>>>>>>>>> can't > >>>>>>>>>>>>> control this part. > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> docker-compose just fits the requirements perfectly that we > >>> need, > >>>>> we > >>>>>>>> have > >>>>>>>>>>>> been using it in the current E2E and turns out it has > >>> advantages > >>>> in > >>>>>>>>>>>> starting many services correctly with health check and > >>> dependency > >>>>>>>>>>>> declaration. (One may say Kubernetes can do the same thing, > >> but > >>>>> itβs > >>>>>>>> kind > >>>>>>>>>>>> of heavy in E2E and GitHub Actions and complex). > >>>>>>>>>>>> > >>>>>>>>>>>> The SkyWalking-CLI wonβt get involved with docker-compose > >>> related > >>>>>>>> things, > >>>>>>>>>>>> it just provides query ability (the same as what it is doing > >>> now, > >>>>>>>> just need > >>>>>>>>>>>> to check whether it covers all the needed query interfaces or > >>>> not), > >>>>>>>> we will > >>>>>>>>>>>> have a dedicated control program to > >>>>>>>>>>>> > >>>>>>>>>>>> (1) start docker-compose (existing standard mechanism, no new > >>>>>>>> knowledge); > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> So, we would provide a script set to drive this? My question > >> is > >>>> more > >>>>>>>> about, > >>>>>>>>>>> how to work with docker-compose, rather than challanging the > >>>> choice. > >>>>>>>>>>> > >>>>>>>>>>> Sheng Wu ε΄ζ > >>>>>>>>>>> Twitter, wusheng1108 > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>>> (2) query actual data (by SkyWalking-CLI, existing repository > >>> and > >>>>>>>> codes); > >>>>>>>>>>>> (3) verify the actual data (by verification tool, a new tool, > >>>>> TODO); > >>>>>>>>>>>> (4) and determine the test result (success or failed); > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> βββββββββ > >>>>>>>>>>>> Zhenxu Ke (ζ―ζ―ζ) > >>>>>>>>>>>> GitHub @kezhenxu94 > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>>> Sheng Wu ε΄ζ > >>>>>>>>>>>>> Twitter, wusheng1108 > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> kezhenxu94@apache <[email protected]> δΊ2020εΉ΄11ζ15ζ₯ε¨ζ₯ > >>>>> δΈε5:27ειοΌ > >>>>>>>>>>>>> > >>>>>>>>>>>>>> Hi the community, I want to raise a discussion to build the > >>>> next > >>>>>>>>>>>>>> generation of E2E test framework for Apache SkyWalking, > >> which > >>>> may > >>>>>>>> not > >>>>>>>>>>>> be a > >>>>>>>>>>>>>> high priority but a necessity. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> As we already know, tests in SkyWalking play a very > >> important > >>>>> role > >>>>>>>> in > >>>>>>>>>>>> the > >>>>>>>>>>>>>> contribution process, and the current E2E tests just work > >>> well > >>>> to > >>>>>>>> verify > >>>>>>>>>>>>>> every single pull request before merging, so why bother to > >>>> build > >>>>>>> the > >>>>>>>>>>>> next > >>>>>>>>>>>>>> generation of E2E test framework? > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> 1. In the SkyWalking's ecosystem, there are many > >> sub-projects > >>>>> that > >>>>>>>> vary > >>>>>>>>>>>> in > >>>>>>>>>>>>>> terms of language, almost all of them need E2E tests to > >> help > >>> us > >>>>>>>> verify > >>>>>>>>>>>> the > >>>>>>>>>>>>>> pull requests, but we can see that they reimplement the E2E > >>>> test > >>>>>>>>>>>> framework > >>>>>>>>>>>>>> in their languages, (or even worse, some of them donβt have > >>> E2E > >>>>>>>> tests at > >>>>>>>>>>>>>> all). Reimplementing the E2E test framework is unnecesarry > >>> and > >>>>>>>>>>>> introduces > >>>>>>>>>>>>>> more duplicated work when adding a new test case. The > >>> ultimate > >>>>>>> goal > >>>>>>>> is > >>>>>>>>>>>> to > >>>>>>>>>>>>>> reuse all the test cases when needed. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> 2. The current E2E framework is built with Java, running > >>>>>>> Java-based > >>>>>>>>>>>>>> program is not an easy thing for non-Java developers > >>>> (maintainers > >>>>>>> of > >>>>>>>>>>>> other > >>>>>>>>>>>>>> sub-projects such as PHP, C++, LUA, etc.), they need to set > >>> up > >>>>> the > >>>>>>>>>>>>>> environment correctly and then run the tests, neither is it > >>> an > >>>>>>> easy > >>>>>>>>>>>> skill > >>>>>>>>>>>>>> to debug the tests. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> 3. Itβs a good opportunity to optimize the E2E tests > >> because > >>> we > >>>>>>>> added > >>>>>>>>>>>> many > >>>>>>>>>>>>>> cases and many duplicated codes that need to be removed. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Therefore we're planning to build an unified, easy-to-use, > >>> and > >>>>>>> fast > >>>>>>>> E2E > >>>>>>>>>>>>>> test framework to benifit all the sub-projects. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Here are some rough ideas about this framework: > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> 0. (Unchanged) We will continue to use docker-compose.yaml > >> to > >>>>>>>>>>>> orchestrate > >>>>>>>>>>>>>> the services that are to be tested, itβs proved to be > >>> friendly > >>>>> and > >>>>>>>>>>>>>> debuggable because we can start it directly even if we are > >>> not > >>>>>>>> writing > >>>>>>>>>>>> E2E > >>>>>>>>>>>>>> tests. > >>>>>>>>>>>>>> 1. This framework will take full advantages of the CLI > >>> project > >>>> to > >>>>>>>> query > >>>>>>>>>>>>>> the traces/metrics/logs, we donβt want to maintain two > >>> versions > >>>>> of > >>>>>>>> query > >>>>>>>>>>>>>> codes anymore, (now we have a version in the test codes and > >>> one > >>>>> in > >>>>>>>> the > >>>>>>>>>>>> CLI). > >>>>>>>>>>>>>> 2. We will build a CLI tool to compare the expected data > >> and > >>>> the > >>>>>>>> actual > >>>>>>>>>>>>>> data, we now have a custom schema of the expected data yaml > >>>> file, > >>>>>>>> the > >>>>>>>>>>>>>> verification codes should be packaged into an executable > >>>> command > >>>>>>> so > >>>>>>>>>>>> that it > >>>>>>>>>>>>>> can be executed standalone without Java and maven > >> knowledge. > >>> I > >>>>>>> hope > >>>>>>>> we > >>>>>>>>>>>> can > >>>>>>>>>>>>>> leverage existing standards to write the YAML files and do > >>>>>>>>>>>> verifications. > >>>>>>>>>>>>>> 3. Build an orchestrating tool to start the > >>>>> `docker-compose.yaml`, > >>>>>>>>>>>> health > >>>>>>>>>>>>>> check, query actual data, and verify the result. > >>>>>>>>>>>>>> 4. (Nice to have) wrap all the tools as a GitHub Actions to > >>>> share > >>>>>>>>>>>> between > >>>>>>>>>>>>>> sub-projects. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> If you have any other better idea or want to complement to > >>> this > >>>>>>>>>>>> proposal, > >>>>>>>>>>>>>> please reply here. I will create an issue to track these > >>> tasks > >>>>>>>> later. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> We have two students (who just finished the Summer Code > >> 2020 > >>>>>>>> Projects) > >>>>>>>>>>>> to > >>>>>>>>>>>>>> lead this part, Ke Zhang [1] and Huaxi Jiang [2], as I said > >>>> this > >>>>>>> is > >>>>>>>> a > >>>>>>>>>>>> not > >>>>>>>>>>>>>> high priority, you have adequate time to get yourselves > >>>> familiar > >>>>>>>> with > >>>>>>>>>>>> how > >>>>>>>>>>>>>> the components (OAP, WebUI, CLI, storages, etc.) of > >>> SkyWalking > >>>>>>> work > >>>>>>>> as > >>>>>>>>>>>> well > >>>>>>>>>>>>>> as the ecosystem. Thanks for your interests and willingness > >>> to > >>>>>>> help. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> [1] https://github.com/Humbertzhang > >>>>>>>>>>>>>> [2] https://github.com/fgksgf/ > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> βββββββββ > >>>>>>>>>>>>>> Zhenxu Ke (ζ―ζ―ζ) > >>>>>>>>>>>>>> GitHub @kezhenxu94 > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> βββββββββ > >>>>>>>>>>>> Zhenxu Ke (ζ―ζ―ζ) > >>>>>>>>>>>> GitHub @kezhenxu94 > >>>>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> βββββββββ > >>>>>>>> Zhenxu Ke (ζ―ζ―ζ) > >>>>>>>> GitHub @kezhenxu94 > >>>>>>>> > >>>>>>>> > >>>>>>> > >>>>> > >>>>> > >>>> > >>> > >> > >
