I agree with the conclusion, and I hope to have some diagrams such as flowcharts or data flow diagrams, which help provide an overview of the E2E test before we start to do this.
Ke Zhang <[email protected]> 于2020年11月26日周四 下午12:34写道： > I think supporting both docker-compose and KIND in the E2E test is > reasonable and necessary, and I also agree with other conclusions.😀 > > kezhenxu94@apache <[email protected]> 于2020年11月26日周四 上午11:58写道： > > > > > > 1. Docker-compose and kind are 2 options to set up the environments. > > > 2. Totally replace the Java and Maven based driving system. > > > 3. Enhance CLI to validate the GraphQL > > > 4. Keep the agent test tool unchanged for now as it is already a > separate > > > system from the e2e. > > > > This conclusion is good for me. > > > > What are others opinions? > > > > ————————— > > Zhenxu Ke (柯振旭) > > GitHub @kezhenxu94 > > > > > On Nov 17, 2020, at 2:42 PM, Sheng Wu <[email protected]> > wrote: > > > > > > Using k8s and docker-compose as 2 options in the test process is > > > reasonable, and should be supported. > > > > > > Let's keep the conclusion as simple as possible. In the next generation > > e2e > > > framework > > > 1. Docker-compose and kind are 2 options to set up the environments. > > > 2. Totally replace the Java and Maven based driving system. > > > 3. Enhance CLI to validate the GraphQL > > > 4. Keep the agent test tool unchanged for now as it is already a > separate > > > system from the e2e. > > > > > > Are these good enough for everyone? > > > > > > Sheng Wu 吴晟 > > > Twitter, wusheng1108 > > > > > > > > > Hongtao Gao <[email protected]> 于2020年11月17日周二 下午12:41写道： > > > > > >>> > > >>> We can see from above, your discussion actually is only related to 3. > > >> > > >> > > >> That's the only I intend to discuss here if you notice my first > > >> response('May SWCK help about provision and demo") in this thread. For > > >> other parts, I don't have any tips to share, though. > > >> Let me explain it more clearly. I want kubernetes to be seen as an > > >> alternative to docker-compose to play a running-engine role in the > test > > >> framework. If kubernetes can't replace > > >> the latter one Is it able to participate in it to solve the dedicated > or > > >> special issues. > > >> > > >> Our e2e test and plugin test frameworks are already top-level > complicity > > >>> thing in the worldwide open source field. Please don't over-design > it, > > >> and > > >>> too aggressive, the world will stay in the hybrid env for a very long > > >> time, > > >>> same as the developer. > > >> > > >> > > >> Reusing current elaborate output might be reasonable. But in this > > thread, > > >> we're talking about the potential solution to build a > > >> next-generation framework, which might mean that the current framework > > >> is too complicated to maintain; we need a more tiny way to support > more > > and > > >> more cases. > > >> > > >> And finally, SWCK is gonna and has to build a similar test framework. > If > > >> the test framework opts for the kubernetes solution, we could share > test > > >> cases and the underlying framework. It's a more efficient > > >> and consistent path for the entire system. > > >> > > >> regards, Hongtao. > > >> > > >> Sheng Wu <[email protected]> 于2020年11月17日周二 上午10:33写道： > > >> > > >>> Hi > > >>> > > >>> I don't think we have to make the decision on k8s or out-of-k8s. > > >>> e2e and all integration tests have their scenarios like SkyWalking > have > > >>> multiple languages based and mesh-based, even Prometheus adoption. > > >>> The same thing happens on the test field, k8s, and out of k8s. > > >>> > > >>> I would like to agree with both of you, k8s is needed, no matter in > > mesh > > >>> solution(ALS, telemetry v2), but also not anyone needs that, such as > > >> agent > > >>> developer. We wouldn't deny the fact, the docker-compose is more > > >>> lightweight and easier to learn. > > >>> So, let me get this straight, for the test you need > > >>> 1. Build the source from different repos > > >>> 2. Build the images > > >>> 3. Use some platform(docker-compose or k8s) to run the images in one > > >> piece. > > >>> 4. Give some inputs to the env > > >>> 5. Use some tools/ways to check what should happen according to (4)'s > > >> input > > >>> 6. Checked or timeout failure. > > >>> > > >>> We can see from above, your discussion actually is only related to 3. > > >> Let's > > >>> not see this as a battle, we need both. Many developers/users use OAP > > and > > >>> SkyWalking out of k8s, that is a simple fact. > > >>> Our e2e test and plugin test frameworks are already top-level > > complicity > > >>> thing in the worldwide open source field. Please don't over-design > it, > > >> and > > >>> too aggressive, the world will stay in the hybrid env for a very long > > >> time, > > >>> same as the developer. > > >>> > > >>> Also, let's keep in mind, k8s is popular becomes it doesn't require > the > > >>> developers to understand as the VM did. We as an open-source > community, > > >> are > > >>> focusing on the developer's capabilities. > > >>> > > >>> Sheng Wu 吴晟 > > >>> Twitter, wusheng1108 > > >>> > > >>> > > >>> Hongtao Gao <[email protected]> 于2020年11月17日周二 上午12:07写道： > > >>> > > >>>>> > > >>>>> but I have been thinking whether or not it is overkill for testing > > >>>>> purposes to introduce Kubernetes things. > > >>>> > > >>>> > > >>>> we all know the fact docker-compose is more portable than > Kubernetes, > > >>>> friendly to local running. But there're also some benefits to > replace > > >> it > > >>>> with kubernetes system: > > >>>> > > >>>> 1. SWCK has to test based on a real Kubernetes environment if the > main > > >>>> repo test framework doesn't support it(following the docker-compose > > >>> stack), > > >>>> That means skywalking ecosystem MUST introduce kubernetes which is > not > > >> an > > >>>> optional component. > > >>>> > > >>>> 2. Kubernetes support to scale applications at runtime, that help > > >>> improve > > >>>> the process of e2e. For instance, we could merge a single instance > and > > >>>> cluster test, by scaling up replicas. This strategy will reduce the > > >> total > > >>>> time sharply. > > >>>> > > >>>> If we are going to use KIND, I hope we can give a script or > something > > >>>>> similar to install all the needed components and create cluster for > > >>>>> convenient testing locally, Ke Zhang and Huaxi. > > >>>> > > >>>> > > >>>> That's SWCK's capacity to provide a standard and simple way to > create > > a > > >>>> cluster in every environment, > > >>>> > > >>>> kezhenxu94@apache <[email protected]> 于2020年11月16日周一 下午7:00写道： > > >>>> > > >>>>> Thanks Hongtao’s suggestion, I’ve thought about adopting > > >>> Kubernetes-based > > >>>>> tech stack (e.g. minikube or KIND), I personally prefer KIND too, > > >> but I > > >>>>> have been thinking whether or not it is overkill for testing > purposes > > >>> to > > >>>>> introduce Kubernetes things. > > >>>>> > > >>>>> Since you mentioned this, we are open to discuss this much more > > >> deeply: > > >>>>> > > >>>>> It would be definitely a benefit to involve more “end”s in the > > >> so-call > > >>>>> “end to end” tests, I’m +1 to leverage SWCK as well. > > >>>>> > > >>>>> If we are going to use KIND, I hope we can give a script or > something > > >>>>> similar to install all the needed components and create cluster for > > >>>>> convenient testing locally, Ke Zhang and Huaxi. > > >>>>> > > >>>>> > > >>>>> ————————— > > >>>>> Zhenxu Ke (柯振旭) > > >>>>> GitHub @kezhenxu94 > > >>>>> > > >>>>>> On Nov 16, 2020, at 4:03 PM, Hongtao Gao <[email protected]> > > >>> wrote: > > >>>>>> > > >>>>>> May SWCK help about provision and demo > > >>>>>> > > >>>>>> 1. SWCK is able to provide a stable and healthy OAP standalone > > >>> instance > > >>>>> or > > >>>>>> cluster for e2e. We could compose indicated CR yamls for different > > >>>>>> scenarios. > > >>>>>> 2. Different demo for different cases is mandatory for e2e and > > >> swck, > > >>> we > > >>>>>> could build a group of demo projects for them. With Kubernetes's > > >>>> support, > > >>>>>> we could > > >>>>>> get free of verbose scripts that are written for installation, > > >>>>> checking > > >>>>>> the state of running applications, collecting logs. > > >>>>>> > > >>>>>> Ppl might have concerts about the minkube we currently have, > > >> which's > > >>>> hard > > >>>>>> to debug and runs slowly. > > >>>>>> > > >>>>>> I prefer to kind[1] instead of minkube. From my experience, kind > > >>> works > > >>>>> fine > > >>>>>> in an e2e scenario(limited cpu, memory resources), and it is run > > >>>> million > > >>>>>> times > > >>>>>> on my team's e2e environment, which proves the stability and > > >>>>>> maintainability of it. > > >>>>>> > > >>>>>> * 1.https://kind.sigs.k8s.io/docs/user/quick-start/ > > >>>>>> > > >>>>>> regards, Hongtao > > >>>>>> > > >>>>>> Sheng Wu <[email protected]> 于2020年11月16日周一 上午8:30写道： > > >>>>>> > > >>>>>>> Inline > > >>>>>>> > > >>>>>>> kezhenxu94@apache <[email protected]> 于2020年11月15日周日 > > >> 下午11:16写道： > > >>>>>>> > > >>>>>>>> Hi Jiajing, > > >>>>>>>> > > >>>>>>>> Thanks for sharing your context and joining the discussion. It’s > > >>> true > > >>>>>>> that > > >>>>>>>> we’re actually tackling the same problems in this domain. Some > of > > >>> the > > >>>>>>>> issues you mentioned are solved in the current E2E framework > > >> (like > > >>>>>>> `retry`, > > >>>>>>>> `timeout`, etc.), while some others are still there without > ideal > > >>>>>>> solution. > > >>>>>>>> Other discussions, please see my comments inline. > > >>>>>>>> > > >>>>>>>>> - the first issue is useful in health check steps, just like > the > > >>>> ones > > >>>>>>>>> defined in readiness probe and liveness probe in kubelet. > > >>>>>>>> > > >>>>>>>> Yes, the reason why we decided to pick docker-compose is that, > it > > >>> has > > >>>>> the > > >>>>>>>> same functionality with kubelet in terms of dependent services > > >>>> startup > > >>>>>>>> order and healthiness checking, while it’s more lite-weight and > > >>> easy > > >>>>> for > > >>>>>>>> both developers and GitHub Actions environment. Right? > > >>>>>>>> > > >>>>>>>>> - the second issue is critical for highly-customized > assertions, > > >>> we > > >>>>>>>>> can leverage the `text/template` module to provide highly > > >>> extensive > > >>>>>>>>> assertion functions > > >>>>>>>> > > >>>>>>>> `text/template` is also the first thought that came to my mind > > >> when > > >>>>>>>> considering customized assertors, customized assertors are for > > >> sure > > >>>>>>>> critical in SkyWalking E2E tests because we have many kinds of > > >>>>>>> assertions, > > >>>>>>>> some of which are not foreseen until we actually need them, > > >>>>>>>> `AtLeastOneGreaterThanZero` is an example. > > >>>>>>>> > > >>>>>>>>> - the last but not the least, the lifecycle hooks can be used > > >> for > > >>>>>>>>> accurate control of test setup and teardown, such as > > >>> `docker-compose > > >>>>>>>>> build, up -d, down` > > >>>>>>>> > > >>>>>>>> The lifecycle hooks are what I didn’t take into consideration > > >>>> because I > > >>>>>>>> thought it is quite natural for test framework to provide this > > >>>>> features, > > >>>>>>>> maybe Ke Zhang and Huaxi can do some research on this part. > > >>>>>>>> > > >>>>>>>>> Together with the aforementioned ideas, we will have a > > >> go-written, > > >>>>>>>>> yaml-data driven e2e testing framework which can overcome the > > >>>> current > > >>>>>>>>> problems that occur, > > >>>>>>>> > > >>>>>>>> I agree. > > >>>>>>>> > > >>>>>>>>> in which the most annoying issue is the fragmentation of > > >> different > > >>>>>>>>> scripts in different submodules, IMO. > > >>>>>>>> > > >>>>>>>> That’s why I list the last item in the previous email, (Nice to > > >>> have, > > >>>>>>> wrap > > >>>>>>>> them into a GitHub Action), with that GitHub Action, we can > > >> simply > > >>>>>>>> configure a workflow/job in every submodule, and the tests are > > >>>> executed > > >>>>>>>> with the shared scripts, no copy-pasting. > > >>>>>>>> > > >>>>>>> > > >>>>>>> Note, GitHub Action will require an Apache release every time > > >>> because > > >>>> we > > >>>>>>> are distributing the binary. > > >>>>>>> > > >>>>>>> > > >>>>>>> Sheng Wu 吴晟 > > >>>>>>> Twitter, wusheng1108 > > >>>>>>> > > >>>>>>> > > >>>>>>>> > > >>>>>>>>> > > >>>>>>>>> Regards, > > >>>>>>>>> Jiajing > > >>>>>>>>> > > >>>>>>>>> On Sun, Nov 15, 2020 at 9:43 PM kezhenxu94@apache < > > >>>>>>> [email protected]> > > >>>>>>>> wrote: > > >>>>>>>>>> > > >>>>>>>>>>> So, we would provide a script set to drive this? > > >>>>>>>>>> > > >>>>>>>>>> Yes, a script or a GoLang-written CLI, it depends. > > >>>>>>>>>> > > >>>>>>>>>>> My question is more about, > > >>>>>>>>>>> how to work with docker-compose > > >>>>>>>>>> > > >>>>>>>>>> It’s the same as what we’re doing now in the current version > of > > >>> E2E > > >>>>>>>> tests, now we have many `docker-compose.yaml` files, and all the > > >>>>>>>> `docker-compose.yaml` files are reusable in the new framework, > we > > >>>> will > > >>>>>>> keep > > >>>>>>>> this mechanism as it is proved to be convenient and easy to use > > >>>>> (remember > > >>>>>>>> when we added an ElasticSearch 7.9 case, and the TiDB storage > E2E > > >>>> test, > > >>>>>>> not > > >>>>>>>> yet merged though, it’s fast/easy to add a new case). > > >>>>>>>>>> > > >>>>>>>>>> ————————— > > >>>>>>>>>> Zhenxu Ke (柯振旭) > > >>>>>>>>>> GitHub @kezhenxu94 > > >>>>>>>>>> > > >>>>>>>>>>> On Nov 15, 2020, at 9:29 PM, Sheng Wu < > > >>> [email protected]> > > >>>>>>>> wrote: > > >>>>>>>>>>> > > >>>>>>>>>>> kezhenxu94@apache <[email protected]> 于2020年11月15日周日 > > >>>> 下午9:16写道： > > >>>>>>>>>>> > > >>>>>>>>>>>> > > >>>>>>>>>>>>> I want to ask, how the docker-compose lands in your mind? I > > >>>> think > > >>>>>>> CLI > > >>>>>>>>>>>> can't > > >>>>>>>>>>>>> control this part. > > >>>>>>>>>>>> > > >>>>>>>>>>>> > > >>>>>>>>>>>> docker-compose just fits the requirements perfectly that we > > >>> need, > > >>>>> we > > >>>>>>>> have > > >>>>>>>>>>>> been using it in the current E2E and turns out it has > > >>> advantages > > >>>> in > > >>>>>>>>>>>> starting many services correctly with health check and > > >>> dependency > > >>>>>>>>>>>> declaration. (One may say Kubernetes can do the same thing, > > >> but > > >>>>> it’s > > >>>>>>>> kind > > >>>>>>>>>>>> of heavy in E2E and GitHub Actions and complex). > > >>>>>>>>>>>> > > >>>>>>>>>>>> The SkyWalking-CLI won’t get involved with docker-compose > > >>> related > > >>>>>>>> things, > > >>>>>>>>>>>> it just provides query ability (the same as what it is doing > > >>> now, > > >>>>>>>> just need > > >>>>>>>>>>>> to check whether it covers all the needed query interfaces > or > > >>>> not), > > >>>>>>>> we will > > >>>>>>>>>>>> have a dedicated control program to > > >>>>>>>>>>>> > > >>>>>>>>>>>> (1) start docker-compose (existing standard mechanism, no > new > > >>>>>>>> knowledge); > > >>>>>>>>>>>> > > >>>>>>>>>>> > > >>>>>>>>>>> So, we would provide a script set to drive this? My question > > >> is > > >>>> more > > >>>>>>>> about, > > >>>>>>>>>>> how to work with docker-compose, rather than challanging the > > >>>> choice. > > >>>>>>>>>>> > > >>>>>>>>>>> Sheng Wu 吴晟 > > >>>>>>>>>>> Twitter, wusheng1108 > > >>>>>>>>>>> > > >>>>>>>>>>> > > >>>>>>>>>>>> (2) query actual data (by SkyWalking-CLI, existing > repository > > >>> and > > >>>>>>>> codes); > > >>>>>>>>>>>> (3) verify the actual data (by verification tool, a new > tool, > > >>>>> TODO); > > >>>>>>>>>>>> (4) and determine the test result (success or failed); > > >>>>>>>>>>>> > > >>>>>>>>>>>> > > >>>>>>>>>>>> ————————— > > >>>>>>>>>>>> Zhenxu Ke (柯振旭) > > >>>>>>>>>>>> GitHub @kezhenxu94 > > >>>>>>>>>>>> > > >>>>>>>>>>>> > > >>>>>>>>>>>> > > >>>>>>>>>>>>> Sheng Wu 吴晟 > > >>>>>>>>>>>>> Twitter, wusheng1108 > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> kezhenxu94@apache <[email protected]> 于2020年11月15日周日 > > >>>>> 下午5:27写道： > > >>>>>>>>>>>>> > > >>>>>>>>>>>>>> Hi the community, I want to raise a discussion to build > the > > >>>> next > > >>>>>>>>>>>>>> generation of E2E test framework for Apache SkyWalking, > > >> which > > >>>> may > > >>>>>>>> not > > >>>>>>>>>>>> be a > > >>>>>>>>>>>>>> high priority but a necessity. > > >>>>>>>>>>>>>> > > >>>>>>>>>>>>>> As we already know, tests in SkyWalking play a very > > >> important > > >>>>> role > > >>>>>>>> in > > >>>>>>>>>>>> the > > >>>>>>>>>>>>>> contribution process, and the current E2E tests just work > > >>> well > > >>>> to > > >>>>>>>> verify > > >>>>>>>>>>>>>> every single pull request before merging, so why bother to > > >>>> build > > >>>>>>> the > > >>>>>>>>>>>> next > > >>>>>>>>>>>>>> generation of E2E test framework? > > >>>>>>>>>>>>>> > > >>>>>>>>>>>>>> 1. In the SkyWalking's ecosystem, there are many > > >> sub-projects > > >>>>> that > > >>>>>>>> vary > > >>>>>>>>>>>> in > > >>>>>>>>>>>>>> terms of language, almost all of them need E2E tests to > > >> help > > >>> us > > >>>>>>>> verify > > >>>>>>>>>>>> the > > >>>>>>>>>>>>>> pull requests, but we can see that they reimplement the > E2E > > >>>> test > > >>>>>>>>>>>> framework > > >>>>>>>>>>>>>> in their languages, (or even worse, some of them don’t > have > > >>> E2E > > >>>>>>>> tests at > > >>>>>>>>>>>>>> all). Reimplementing the E2E test framework is unnecesarry > > >>> and > > >>>>>>>>>>>> introduces > > >>>>>>>>>>>>>> more duplicated work when adding a new test case. The > > >>> ultimate > > >>>>>>> goal > > >>>>>>>> is > > >>>>>>>>>>>> to > > >>>>>>>>>>>>>> reuse all the test cases when needed. > > >>>>>>>>>>>>>> > > >>>>>>>>>>>>>> 2. The current E2E framework is built with Java, running > > >>>>>>> Java-based > > >>>>>>>>>>>>>> program is not an easy thing for non-Java developers > > >>>> (maintainers > > >>>>>>> of > > >>>>>>>>>>>> other > > >>>>>>>>>>>>>> sub-projects such as PHP, C++, LUA, etc.), they need to > set > > >>> up > > >>>>> the > > >>>>>>>>>>>>>> environment correctly and then run the tests, neither is > it > > >>> an > > >>>>>>> easy > > >>>>>>>>>>>> skill > > >>>>>>>>>>>>>> to debug the tests. > > >>>>>>>>>>>>>> > > >>>>>>>>>>>>>> 3. It’s a good opportunity to optimize the E2E tests > > >> because > > >>> we > > >>>>>>>> added > > >>>>>>>>>>>> many > > >>>>>>>>>>>>>> cases and many duplicated codes that need to be removed. > > >>>>>>>>>>>>>> > > >>>>>>>>>>>>>> Therefore we're planning to build an unified, easy-to-use, > > >>> and > > >>>>>>> fast > > >>>>>>>> E2E > > >>>>>>>>>>>>>> test framework to benifit all the sub-projects. > > >>>>>>>>>>>>>> > > >>>>>>>>>>>>>> Here are some rough ideas about this framework: > > >>>>>>>>>>>>>> > > >>>>>>>>>>>>>> 0. (Unchanged) We will continue to use docker-compose.yaml > > >> to > > >>>>>>>>>>>> orchestrate > > >>>>>>>>>>>>>> the services that are to be tested, it’s proved to be > > >>> friendly > > >>>>> and > > >>>>>>>>>>>>>> debuggable because we can start it directly even if we are > > >>> not > > >>>>>>>> writing > > >>>>>>>>>>>> E2E > > >>>>>>>>>>>>>> tests. > > >>>>>>>>>>>>>> 1. This framework will take full advantages of the CLI > > >>> project > > >>>> to > > >>>>>>>> query > > >>>>>>>>>>>>>> the traces/metrics/logs, we don’t want to maintain two > > >>> versions > > >>>>> of > > >>>>>>>> query > > >>>>>>>>>>>>>> codes anymore, (now we have a version in the test codes > and > > >>> one > > >>>>> in > > >>>>>>>> the > > >>>>>>>>>>>> CLI). > > >>>>>>>>>>>>>> 2. We will build a CLI tool to compare the expected data > > >> and > > >>>> the > > >>>>>>>> actual > > >>>>>>>>>>>>>> data, we now have a custom schema of the expected data > yaml > > >>>> file, > > >>>>>>>> the > > >>>>>>>>>>>>>> verification codes should be packaged into an executable > > >>>> command > > >>>>>>> so > > >>>>>>>>>>>> that it > > >>>>>>>>>>>>>> can be executed standalone without Java and maven > > >> knowledge. > > >>> I > > >>>>>>> hope > > >>>>>>>> we > > >>>>>>>>>>>> can > > >>>>>>>>>>>>>> leverage existing standards to write the YAML files and do > > >>>>>>>>>>>> verifications. > > >>>>>>>>>>>>>> 3. Build an orchestrating tool to start the > > >>>>> `docker-compose.yaml`, > > >>>>>>>>>>>> health > > >>>>>>>>>>>>>> check, query actual data, and verify the result. > > >>>>>>>>>>>>>> 4. (Nice to have) wrap all the tools as a GitHub Actions > to > > >>>> share > > >>>>>>>>>>>> between > > >>>>>>>>>>>>>> sub-projects. > > >>>>>>>>>>>>>> > > >>>>>>>>>>>>>> If you have any other better idea or want to complement to > > >>> this > > >>>>>>>>>>>> proposal, > > >>>>>>>>>>>>>> please reply here. I will create an issue to track these > > >>> tasks > > >>>>>>>> later. > > >>>>>>>>>>>>>> > > >>>>>>>>>>>>>> We have two students (who just finished the Summer Code > > >> 2020 > > >>>>>>>> Projects) > > >>>>>>>>>>>> to > > >>>>>>>>>>>>>> lead this part, Ke Zhang [1] and Huaxi Jiang [2], as I > said > > >>>> this > > >>>>>>> is > > >>>>>>>> a > > >>>>>>>>>>>> not > > >>>>>>>>>>>>>> high priority, you have adequate time to get yourselves > > >>>> familiar > > >>>>>>>> with > > >>>>>>>>>>>> how > > >>>>>>>>>>>>>> the components (OAP, WebUI, CLI, storages, etc.) of > > >>> SkyWalking > > >>>>>>> work > > >>>>>>>> as > > >>>>>>>>>>>> well > > >>>>>>>>>>>>>> as the ecosystem. Thanks for your interests and > willingness > > >>> to > > >>>>>>> help. > > >>>>>>>>>>>>>> > > >>>>>>>>>>>>>> [1] https://github.com/Humbertzhang > > >>>>>>>>>>>>>> [2] https://github.com/fgksgf/ > > >>>>>>>>>>>>>> > > >>>>>>>>>>>>>> ————————— > > >>>>>>>>>>>>>> Zhenxu Ke (柯振旭) > > >>>>>>>>>>>>>> GitHub @kezhenxu94 > > >>>>>>>>>>>>>> > > >>>>>>>>>>>>>> > > >>>>>>>>>>>> > > >>>>>>>>>>>> ————————— > > >>>>>>>>>>>> Zhenxu Ke (柯振旭) > > >>>>>>>>>>>> GitHub @kezhenxu94 > > >>>>>>>>>> > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> ————————— > > >>>>>>>> Zhenxu Ke (柯振旭) > > >>>>>>>> GitHub @kezhenxu94 > > >>>>>>>> > > >>>>>>>> > > >>>>>>> > > >>>>> > > >>>>> > > >>>> > > >>> > > >> > > > > >
