Big +1 to java-based e2e tests. It'll be much easier to write/debug these tests.
On Wed, Nov 18, 2020 at 9:44 PM Leonard Xu <xbjt...@gmail.com> wrote: > +1 to stop using bash scripts, > and I also have experienced the bash scripts that is really hard to > maintain and debug, thanks @Robert for the great work again. > > I think testcontainers is a nice candidate. > > Best, > Leonard > > > 在 2020年11月18日，19:46，Aljoscha Krettek <aljos...@apache.org> 写道： > > > > +1 > > > > And I want to second Arvid's mention of testcontainers [1]. > > > > [1] https://www.testcontainers.org/ > > > > On 18.11.20 10:43, Yang Wang wrote: > >> Thanks till and Jark for sharing the information. > >> I am also +1 for this proposal and glad to wire the new introduced K8s > HA > >> e2e tests to java based framework. > >> Best, > >> Yang > >> Jark Wu <imj...@gmail.com> 于2020年11月18日周三 下午5:23写道： > >>> +1 to use the Java-based testing framework and +1 for using docker > images > >>> in the future. > >>> IIUC, the Java-based testing framework refers to the > >>> `flink-end-to-end-tests-common` module. > >>> The java-based framework helped us a lot when debugging the unstable > e2e > >>> tests. > >>> > >>> Best, > >>> Jark > >>> > >>> On Wed, 18 Nov 2020 at 14:42, Yang Wang <danrtsey...@gmail.com> wrote: > >>> > >>>> Thanks for starting this discussion. > >>>> > >>>> In general, I agree with you that a java-based testing framework is > >>> better > >>>> than the bash-based. It will > >>>> help a lot for the commons and utilities. > >>>> > >>>> Since I am trying to add a new bash-based Kubernetes HA test, I have > some > >>>> quick questions. > >>>> * I am not sure where the java-based framework is. Do you mean > >>>> "flink-jepsen" module or sth else? > >>>> * Maybe it will be harder to run a cli command(e.g. flink run / > >>>> run-application) to submit a Flink job in the java-based framework. > >>>> * It will be harder to inject some operations. For example, kill the > >>>> JobManager in Kubernetes. Currently, I > >>>> am trying to use "kubectl exec" to do this. > >>>> > >>>> > >>>> Best, > >>>> Yang > >>>> > >>>> Robert Metzger <rmetz...@apache.org> 于2020年11月17日周二 下午11:36写道： > >>>> > >>>>> Hi all, > >>>>> > >>>>> Since we are currently testing the 1.12 release, and potentially > adding > >>>>> more automated e2e tests, I would like to bring up our end to end > tests > >>>> for > >>>>> discussion. > >>>>> > >>>>> Some time ago, we introduced a Java-based testing framework, with the > >>>> idea > >>>>> of replacing the current bash-based end to end tests. > >>>>> Since the introduction of the java-based framework, more bash tests > >>> were > >>>>> actually added, making a future migration even harder. > >>>>> > >>>>> *For that reason, I would like to propose that we are stopping to add > >>> any > >>>>> new bash end to end tests to Flink. All new end to end tests must be > >>>>> written in Java and rely on the existing testing framework.* > >>>>> > >>>>> For the 1.13 release, I'm trying to find some time to revisit > potential > >>>>> improvements for the existing java e2e framework (such as using > Docker > >>>>> images everywhere), as well as a migration plan for the existing bash > >>>>> tests. We have a large number of bash e2e tests that are just > >>>> parameterized > >>>>> differently. If we would start migrating them to Java, we could move > a > >>>>> larger proportion of tests over to the new Java framework, and tackle > >>> the > >>>>> more involved bash tests later (kerberized yarn, kubernetes, ...). > >>>>> > >>>>> Let me know what you think! > >>>>> > >>>>> Best, > >>>>> Robert > >>>>> > >>>>> > >>>>> PS: If you are wondering why I'm bringing this up now: I'm spending > >>>> quite a > >>>>> lot of time trying to figure out really hard to debug issues with our > >>>> bash > >>>>> testing infra. > >>>>> Also, it is very difficult to introduce something generic for all > tests > >>>>> (such as a test-timeout, using docker as the preferred deployment > >>> method > >>>>> etc.) since the tests often don't share common tooling. > >>>>> Speaking about tooling: there are a lot of utilities everywhere, > >>>> sometimes > >>>>> duplicated, with different features / stability etc. > >>>>> I believe bash is not the right tool for a project this size (in > terms > >>> of > >>>>> developers and lines of code) > >>>>> > >>>> > >>> > > > > -- Best regards! Rui Li
