Hi I don't think we have to make the decision on k8s or out-of-k8s. e2e and all integration tests have their scenarios like SkyWalking have multiple languages based and mesh-based, even Prometheus adoption. The same thing happens on the test field, k8s, and out of k8s.
I would like to agree with both of you, k8s is needed, no matter in mesh solution(ALS, telemetry v2), but also not anyone needs that, such as agent developer. We wouldn't deny the fact, the docker-compose is more lightweight and easier to learn. So, let me get this straight, for the test you need 1. Build the source from different repos 2. Build the images 3. Use some platform(docker-compose or k8s) to run the images in one piece. 4. Give some inputs to the env 5. Use some tools/ways to check what should happen according to (4)'s input 6. Checked or timeout failure. We can see from above, your discussion actually is only related to 3. Let's not see this as a battle, we need both. Many developers/users use OAP and SkyWalking out of k8s, that is a simple fact. Our e2e test and plugin test frameworks are already top-level complicity thing in the worldwide open source field. Please don't over-design it, and too aggressive, the world will stay in the hybrid env for a very long time, same as the developer. Also, let's keep in mind, k8s is popular becomes it doesn't require the developers to understand as the VM did. We as an open-source community, are focusing on the developer's capabilities. Sheng Wu 吴晟 Twitter, wusheng1108 Hongtao Gao <[email protected]> 于2020年11月17日周二 上午12:07写道： > > > > but I have been thinking whether or not it is overkill for testing > > purposes to introduce Kubernetes things. > > > we all know the fact docker-compose is more portable than Kubernetes, > friendly to local running. But there're also some benefits to replace it > with kubernetes system: > > 1. SWCK has to test based on a real Kubernetes environment if the main > repo test framework doesn't support it(following the docker-compose stack), > That means skywalking ecosystem MUST introduce kubernetes which is not an > optional component. > > 2. Kubernetes support to scale applications at runtime, that help improve > the process of e2e. For instance, we could merge a single instance and > cluster test, by scaling up replicas. This strategy will reduce the total > time sharply. > > If we are going to use KIND, I hope we can give a script or something > > similar to install all the needed components and create cluster for > > convenient testing locally, Ke Zhang and Huaxi. > > > That's SWCK's capacity to provide a standard and simple way to create a > cluster in every environment, > > kezhenxu94@apache <[email protected]> 于2020年11月16日周一 下午7:00写道： > > > Thanks Hongtao’s suggestion, I’ve thought about adopting Kubernetes-based > > tech stack (e.g. minikube or KIND), I personally prefer KIND too, but I > > have been thinking whether or not it is overkill for testing purposes to > > introduce Kubernetes things. > > > > Since you mentioned this, we are open to discuss this much more deeply: > > > > It would be definitely a benefit to involve more “end”s in the so-call > > “end to end” tests, I’m +1 to leverage SWCK as well. > > > > If we are going to use KIND, I hope we can give a script or something > > similar to install all the needed components and create cluster for > > convenient testing locally, Ke Zhang and Huaxi. > > > > > > ————————— > > Zhenxu Ke (柯振旭) > > GitHub @kezhenxu94 > > > > > On Nov 16, 2020, at 4:03 PM, Hongtao Gao <[email protected]> wrote: > > > > > > May SWCK help about provision and demo > > > > > > 1. SWCK is able to provide a stable and healthy OAP standalone instance > > or > > > cluster for e2e. We could compose indicated CR yamls for different > > > scenarios. > > > 2. Different demo for different cases is mandatory for e2e and swck, we > > > could build a group of demo projects for them. With Kubernetes's > support, > > > we could > > > get free of verbose scripts that are written for installation, > > checking > > > the state of running applications, collecting logs. > > > > > > Ppl might have concerts about the minkube we currently have, which's > hard > > > to debug and runs slowly. > > > > > > I prefer to kind[1] instead of minkube. From my experience, kind works > > fine > > > in an e2e scenario(limited cpu, memory resources), and it is run > million > > > times > > > on my team's e2e environment, which proves the stability and > > > maintainability of it. > > > > > > * 1.https://kind.sigs.k8s.io/docs/user/quick-start/ > > > > > > regards, Hongtao > > > > > > Sheng Wu <[email protected]> 于2020年11月16日周一 上午8:30写道： > > > > > >> Inline > > >> > > >> kezhenxu94@apache <[email protected]> 于2020年11月15日周日 下午11:16写道： > > >> > > >>> Hi Jiajing, > > >>> > > >>> Thanks for sharing your context and joining the discussion. It’s true > > >> that > > >>> we’re actually tackling the same problems in this domain. Some of the > > >>> issues you mentioned are solved in the current E2E framework (like > > >> `retry`, > > >>> `timeout`, etc.), while some others are still there without ideal > > >> solution. > > >>> Other discussions, please see my comments inline. > > >>> > > >>>> - the first issue is useful in health check steps, just like the > ones > > >>>> defined in readiness probe and liveness probe in kubelet. > > >>> > > >>> Yes, the reason why we decided to pick docker-compose is that, it has > > the > > >>> same functionality with kubelet in terms of dependent services > startup > > >>> order and healthiness checking, while it’s more lite-weight and easy > > for > > >>> both developers and GitHub Actions environment. Right? > > >>> > > >>>> - the second issue is critical for highly-customized assertions, we > > >>>> can leverage the `text/template` module to provide highly extensive > > >>>> assertion functions > > >>> > > >>> `text/template` is also the first thought that came to my mind when > > >>> considering customized assertors, customized assertors are for sure > > >>> critical in SkyWalking E2E tests because we have many kinds of > > >> assertions, > > >>> some of which are not foreseen until we actually need them, > > >>> `AtLeastOneGreaterThanZero` is an example. > > >>> > > >>>> - the last but not the least, the lifecycle hooks can be used for > > >>>> accurate control of test setup and teardown, such as `docker-compose > > >>>> build, up -d, down` > > >>> > > >>> The lifecycle hooks are what I didn’t take into consideration > because I > > >>> thought it is quite natural for test framework to provide this > > features, > > >>> maybe Ke Zhang and Huaxi can do some research on this part. > > >>> > > >>>> Together with the aforementioned ideas, we will have a go-written, > > >>>> yaml-data driven e2e testing framework which can overcome the > current > > >>>> problems that occur, > > >>> > > >>> I agree. > > >>> > > >>>> in which the most annoying issue is the fragmentation of different > > >>>> scripts in different submodules, IMO. > > >>> > > >>> That’s why I list the last item in the previous email, (Nice to have, > > >> wrap > > >>> them into a GitHub Action), with that GitHub Action, we can simply > > >>> configure a workflow/job in every submodule, and the tests are > executed > > >>> with the shared scripts, no copy-pasting. > > >>> > > >> > > >> Note, GitHub Action will require an Apache release every time because > we > > >> are distributing the binary. > > >> > > >> > > >> Sheng Wu 吴晟 > > >> Twitter, wusheng1108 > > >> > > >> > > >>> > > >>>> > > >>>> Regards, > > >>>> Jiajing > > >>>> > > >>>> On Sun, Nov 15, 2020 at 9:43 PM kezhenxu94@apache < > > >> [email protected]> > > >>> wrote: > > >>>>> > > >>>>>> So, we would provide a script set to drive this? > > >>>>> > > >>>>> Yes, a script or a GoLang-written CLI, it depends. > > >>>>> > > >>>>>> My question is more about, > > >>>>>> how to work with docker-compose > > >>>>> > > >>>>> It’s the same as what we’re doing now in the current version of E2E > > >>> tests, now we have many `docker-compose.yaml` files, and all the > > >>> `docker-compose.yaml` files are reusable in the new framework, we > will > > >> keep > > >>> this mechanism as it is proved to be convenient and easy to use > > (remember > > >>> when we added an ElasticSearch 7.9 case, and the TiDB storage E2E > test, > > >> not > > >>> yet merged though, it’s fast/easy to add a new case). > > >>>>> > > >>>>> ————————— > > >>>>> Zhenxu Ke (柯振旭) > > >>>>> GitHub @kezhenxu94 > > >>>>> > > >>>>>> On Nov 15, 2020, at 9:29 PM, Sheng Wu <[email protected]> > > >>> wrote: > > >>>>>> > > >>>>>> kezhenxu94@apache <[email protected]> 于2020年11月15日周日 > 下午9:16写道： > > >>>>>> > > >>>>>>> > > >>>>>>>> I want to ask, how the docker-compose lands in your mind? I > think > > >> CLI > > >>>>>>> can't > > >>>>>>>> control this part. > > >>>>>>> > > >>>>>>> > > >>>>>>> docker-compose just fits the requirements perfectly that we need, > > we > > >>> have > > >>>>>>> been using it in the current E2E and turns out it has advantages > in > > >>>>>>> starting many services correctly with health check and dependency > > >>>>>>> declaration. (One may say Kubernetes can do the same thing, but > > it’s > > >>> kind > > >>>>>>> of heavy in E2E and GitHub Actions and complex). > > >>>>>>> > > >>>>>>> The SkyWalking-CLI won’t get involved with docker-compose related > > >>> things, > > >>>>>>> it just provides query ability (the same as what it is doing now, > > >>> just need > > >>>>>>> to check whether it covers all the needed query interfaces or > not), > > >>> we will > > >>>>>>> have a dedicated control program to > > >>>>>>> > > >>>>>>> (1) start docker-compose (existing standard mechanism, no new > > >>> knowledge); > > >>>>>>> > > >>>>>> > > >>>>>> So, we would provide a script set to drive this? My question is > more > > >>> about, > > >>>>>> how to work with docker-compose, rather than challanging the > choice. > > >>>>>> > > >>>>>> Sheng Wu 吴晟 > > >>>>>> Twitter, wusheng1108 > > >>>>>> > > >>>>>> > > >>>>>>> (2) query actual data (by SkyWalking-CLI, existing repository and > > >>> codes); > > >>>>>>> (3) verify the actual data (by verification tool, a new tool, > > TODO); > > >>>>>>> (4) and determine the test result (success or failed); > > >>>>>>> > > >>>>>>> > > >>>>>>> ————————— > > >>>>>>> Zhenxu Ke (柯振旭) > > >>>>>>> GitHub @kezhenxu94 > > >>>>>>> > > >>>>>>> > > >>>>>>> > > >>>>>>>> Sheng Wu 吴晟 > > >>>>>>>> Twitter, wusheng1108 > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> kezhenxu94@apache <[email protected]> 于2020年11月15日周日 > > 下午5:27写道： > > >>>>>>>> > > >>>>>>>>> Hi the community, I want to raise a discussion to build the > next > > >>>>>>>>> generation of E2E test framework for Apache SkyWalking, which > may > > >>> not > > >>>>>>> be a > > >>>>>>>>> high priority but a necessity. > > >>>>>>>>> > > >>>>>>>>> As we already know, tests in SkyWalking play a very important > > role > > >>> in > > >>>>>>> the > > >>>>>>>>> contribution process, and the current E2E tests just work well > to > > >>> verify > > >>>>>>>>> every single pull request before merging, so why bother to > build > > >> the > > >>>>>>> next > > >>>>>>>>> generation of E2E test framework? > > >>>>>>>>> > > >>>>>>>>> 1. In the SkyWalking's ecosystem, there are many sub-projects > > that > > >>> vary > > >>>>>>> in > > >>>>>>>>> terms of language, almost all of them need E2E tests to help us > > >>> verify > > >>>>>>> the > > >>>>>>>>> pull requests, but we can see that they reimplement the E2E > test > > >>>>>>> framework > > >>>>>>>>> in their languages, (or even worse, some of them don’t have E2E > > >>> tests at > > >>>>>>>>> all). Reimplementing the E2E test framework is unnecesarry and > > >>>>>>> introduces > > >>>>>>>>> more duplicated work when adding a new test case. The ultimate > > >> goal > > >>> is > > >>>>>>> to > > >>>>>>>>> reuse all the test cases when needed. > > >>>>>>>>> > > >>>>>>>>> 2. The current E2E framework is built with Java, running > > >> Java-based > > >>>>>>>>> program is not an easy thing for non-Java developers > (maintainers > > >> of > > >>>>>>> other > > >>>>>>>>> sub-projects such as PHP, C++, LUA, etc.), they need to set up > > the > > >>>>>>>>> environment correctly and then run the tests, neither is it an > > >> easy > > >>>>>>> skill > > >>>>>>>>> to debug the tests. > > >>>>>>>>> > > >>>>>>>>> 3. It’s a good opportunity to optimize the E2E tests because we > > >>> added > > >>>>>>> many > > >>>>>>>>> cases and many duplicated codes that need to be removed. > > >>>>>>>>> > > >>>>>>>>> Therefore we're planning to build an unified, easy-to-use, and > > >> fast > > >>> E2E > > >>>>>>>>> test framework to benifit all the sub-projects. > > >>>>>>>>> > > >>>>>>>>> Here are some rough ideas about this framework: > > >>>>>>>>> > > >>>>>>>>> 0. (Unchanged) We will continue to use docker-compose.yaml to > > >>>>>>> orchestrate > > >>>>>>>>> the services that are to be tested, it’s proved to be friendly > > and > > >>>>>>>>> debuggable because we can start it directly even if we are not > > >>> writing > > >>>>>>> E2E > > >>>>>>>>> tests. > > >>>>>>>>> 1. This framework will take full advantages of the CLI project > to > > >>> query > > >>>>>>>>> the traces/metrics/logs, we don’t want to maintain two versions > > of > > >>> query > > >>>>>>>>> codes anymore, (now we have a version in the test codes and one > > in > > >>> the > > >>>>>>> CLI). > > >>>>>>>>> 2. We will build a CLI tool to compare the expected data and > the > > >>> actual > > >>>>>>>>> data, we now have a custom schema of the expected data yaml > file, > > >>> the > > >>>>>>>>> verification codes should be packaged into an executable > command > > >> so > > >>>>>>> that it > > >>>>>>>>> can be executed standalone without Java and maven knowledge. I > > >> hope > > >>> we > > >>>>>>> can > > >>>>>>>>> leverage existing standards to write the YAML files and do > > >>>>>>> verifications. > > >>>>>>>>> 3. Build an orchestrating tool to start the > > `docker-compose.yaml`, > > >>>>>>> health > > >>>>>>>>> check, query actual data, and verify the result. > > >>>>>>>>> 4. (Nice to have) wrap all the tools as a GitHub Actions to > share > > >>>>>>> between > > >>>>>>>>> sub-projects. > > >>>>>>>>> > > >>>>>>>>> If you have any other better idea or want to complement to this > > >>>>>>> proposal, > > >>>>>>>>> please reply here. I will create an issue to track these tasks > > >>> later. > > >>>>>>>>> > > >>>>>>>>> We have two students (who just finished the Summer Code 2020 > > >>> Projects) > > >>>>>>> to > > >>>>>>>>> lead this part, Ke Zhang [1] and Huaxi Jiang [2], as I said > this > > >> is > > >>> a > > >>>>>>> not > > >>>>>>>>> high priority, you have adequate time to get yourselves > familiar > > >>> with > > >>>>>>> how > > >>>>>>>>> the components (OAP, WebUI, CLI, storages, etc.) of SkyWalking > > >> work > > >>> as > > >>>>>>> well > > >>>>>>>>> as the ecosystem. Thanks for your interests and willingness to > > >> help. > > >>>>>>>>> > > >>>>>>>>> [1] https://github.com/Humbertzhang > > >>>>>>>>> [2] https://github.com/fgksgf/ > > >>>>>>>>> > > >>>>>>>>> ————————— > > >>>>>>>>> Zhenxu Ke (柯振旭) > > >>>>>>>>> GitHub @kezhenxu94 > > >>>>>>>>> > > >>>>>>>>> > > >>>>>>> > > >>>>>>> ————————— > > >>>>>>> Zhenxu Ke (柯振旭) > > >>>>>>> GitHub @kezhenxu94 > > >>>>> > > >>> > > >>> > > >>> > > >>> ————————— > > >>> Zhenxu Ke (柯振旭) > > >>> GitHub @kezhenxu94 > > >>> > > >>> > > >> > > > > >
