>From a user perspective it would be nice if it is possible to use Junit 5 also for integration tests using MiniClusterWithClientResource [1].
I would be happy to help you with the migration of a few modules, if you need a hand with it. Regards, Jörn [1] https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/dev/datastream/testing/#testing-flink-jobs On Tue, May 25, 2021 at 10:33 AM Till Rohrmann <trohrm...@apache.org> wrote: > > Thanks for joining the discussion Qingsheng. In general, I am not opposed > to upgrading our testing library to JUnit 5. Also, the idea of starting > with individual modules and do it incrementally sounds reasonable. > > However, before starting to do it like this, the community should agree > that we want to replace JUnit 4 with JUnit 5 eventually. This does not mean > to rewrite all existing tests to use JUnit 5 but at least the default for > all new tests should be JUnit 5 at some point. Otherwise, I fear that we > will fragment the project into modules that use JUnit 5 and those that use > JUnit 4. If this happens then it becomes harder for people to work on the > code base because they always need to know which testing library to use in > which module. > > Cheers, > Till > > On Tue, May 25, 2021 at 8:53 AM Qingsheng Ren <renqs...@gmail.com> wrote: > > > Hi forks, > > > > I’d like to resume the discussion on migrating to JUnit 5. I’ve been > > working on a connector testing framework and recently have an exploration > > on JUnit 5. I think some features are very helpful for the development of > > the testing framework: > > > > • Extensions > > > > JUnit 5 introduces a new Extension model, which provide a pluggable > > mechanism for extending test classes, like managing test lifecycles and > > providing parameters. Also with the help of extension, we can get rid of > > some limitations introduced by class inheritance, like current TestLogger & > > KafkaTestBase. In testing framework this is helpful for handling the > > lifecycle of Flink cluster and external system. > > > > • Annotations > > > > JUnit 5 provides better support in annotations, working together with > > extensions. We can simple mark types/fields/methods in the test, then let > > extension to search these elements and manage their lifecycle in the test. > > For example test with annotation @MiniCluster will be provided with a > > lifecycle-managed MiniCluster automatically. > > > > • Parameterized tests > > > > JUnit 5 supports more powerful parameterized tests. Testing framework uses > > this to inject different test environments and external contexts into the > > same test case, to run the case under different scenarios. > > > > So I think JUnit 5 is quite flexible for developing such a framework or > > test utility based on it. My suggestion is that we can take connector > > testing framework as a starting point of using JUnit 5, then we can expand > > our exploration to more modules, finally dive into the entire project. > > > > -- > > Best Regards, > > > > Qingsheng Ren > > Email: renqs...@gmail.com > > On Dec 1, 2020, 4:54 PM +0800, Khachatryan Roman , wrote: > > > +1 for the migration > > > > > > (I agree with Dawid, for me the most important benefit is better support > > of > > > parameterized tests). > > > > > > Regards, > > > Roman > > > > > > > > > On Mon, Nov 30, 2020 at 9:42 PM Arvid Heise <ar...@ververica.com> wrote: > > > > > > > Hi Till, > > > > > > > > immediate benefit would be mostly nested tests for a better test > > structure > > > > and new parameterized tests for less clutter (often test functionality > > is > > > > split into parameterized test and non-parameterized test because of > > JUnit4 > > > > limitation). Additionally, having Java8 lambdas to perform fine-grain > > > > exception handling would make all related tests more readable (@Test > > only > > > > allows one exception per test method, while in reality we often have > > more > > > > exceptions / more fine grain assertions and need to resort to > > try-catch -- > > > > yuck!). The extension mechanism would also make the mini cluster much > > > > easier to use: we often have to start the cluster manually because of > > > > test-specific configuration, which can be easily avoided in JUnit5. > > > > > > > > In the medium and long-term, I'd also like to use the modular > > > > infrastructure and improved parallelization. The former would allow us > > > > better to implement cross-cutting features like TestLogger (why do we > > need > > > > to extend that manually in every test?). The latter is more relevant > > for > > > > the next push on CI, which would be especially interesting with e2e > > being > > > > available in Java. > > > > > > > > On Mon, Nov 30, 2020 at 2:07 PM Dawid Wysakowicz < > > dwysakow...@apache.org> > > > > wrote: > > > > > > > > > Hi all, > > > > > > > > > > Just wanted to express my support for the idea. I did miss certain > > > > > features of JUnit 5 already, an important one being much better > > support > > > > > for parameterized tests. > > > > > > > > > > Best, > > > > > > > > > > Dawid > > > > > > > > > > On 30/11/2020 13:50, Arvid Heise wrote: > > > > > > Hi Chesnay, > > > > > > > > > > > > The vintage runner supports the old annotations, so we don't have > > to > > > > > change > > > > > > them in the first step. > > > > > > > > > > > > The only thing that we need to change are all rules that do not > > extend > > > > > > ExternalResource (e.g., TestWatcher used in TestLogger). This > > change > > > > > needs > > > > > > to be done swiftly as this affects the shared infrastructure as you > > > > have > > > > > > mentioned. > > > > > > > > > > > > Only afterwards, we start to actually migrate the individual tests. > > > > That > > > > > > can be done module by module or as we go. I actually found a nice > > > > article > > > > > > that leverages the migration assist of IntelliJ [1]. > > > > > > > > > > > > As the last stop, we remove the vintage runner - all JUnit4 tests > > have > > > > > been > > > > > > migrated and new tests cannot use old annotation etc. anymore. > > > > > > > > > > > > [1] > > > > > > > > > > > > > > > > > https://blog.jetbrains.com/idea/2020/08/migrating-from-junit-4-to-junit-5/ > > > > > > > > > > > > On Mon, Nov 30, 2020 at 1:32 PM Chesnay Schepler < > > ches...@apache.org> > > > > > wrote: > > > > > > > > > > > > > I presume we cannot do the migration module-wise due to shared > > test > > > > > > > utilities that rely on JUnit interfaces? > > > > > > > > > > > > > > On 11/30/2020 1:30 PM, Chesnay Schepler wrote: > > > > > > > > Is it feasible that 2 people can do the migration within a > > short > > > > > > > > time-frame (say, a week)? > > > > > > > > Must the migration of a test be done in one go, or can we for > > example > > > > > > > > first rename all the Before/After annotations and then to the > > rest? > > > > > > > > Are there any issues with other test dependencies (i.e., > > hamcrest, > > > > > > > > powermock (PowerMockRunner), mockito) that we should be aware > > of? > > > > > > > > > > > > > > > > I generally like the idea of using JUnit 5, but am wary of this > > > > > > > > migration dragging on for too long. > > > > > > > > > > > > > > > > On 11/27/2020 3:29 PM, Arvid Heise wrote: > > > > > > > > > Dear devs, > > > > > > > > > > > > > > > > > > I'd like to start a discussion to migrate to a higher JUnit > > version. > > > > > > > > > > > > > > > > > > The main motivations are: > > > > > > > > > - Making full use of Java 8 Lambdas for writing easier to > > read tests > > > > > > > > > and a > > > > > > > > > better performing way of composing failure messages. > > > > > > > > > - Improved test structures with nested and dynamic tests. > > > > > > > > > - Much better support for parameterized tests to avoid > > separating > > > > > > > > > parameterized and non-parameterized parts into different test > > > > classes. > > > > > > > > > - Composable dependencies and better hooks for advanced use > > cases > > > > > > > > > (TestLogger). > > > > > > > > > - Better exception verification > > > > > > > > > - More current infrastructure > > > > > > > > > - Better parallelizable > > > > > > > > > > > > > > > > > > Why now? > > > > > > > > > - JUnit5 is now mature enough to consider it for such a > > complex > > > > > project > > > > > > > > > - We are porting more and more e2e tests to JUnit and it > > would be a > > > > > > > > > pity to > > > > > > > > > do all the work twice (okay some already has been done and > > would > > > > > > > > > result in > > > > > > > > > adjustments, but the sooner we migrate, the less needs to be > > touched > > > > > > > > > twice) > > > > > > > > > > > > > > > > > > Why JUnit5? > > > > > > > > > There are other interesting alternatives, such as TestNG. > > I'm happy > > > > > > > > > to hear > > > > > > > > > specific alternatives. For now, I'd like to focus on JUnit4 > > for an > > > > > > > > > easier > > > > > > > > > migration path. > > > > > > > > > > > > > > > > > > Please discuss if you would also be interested in moving > > onward. To > > > > > get > > > > > > > > > some overview, I'd like to see some informal +1 for the > > options: > > > > > > > > > > > > > > > > > > [ ] Stick to JUnit4 for the time being > > > > > > > > > [ ] Move to JUnit5 (see migration path below) > > > > > > > > > [ ] Alternative idea + advantages over JUnit5 + some very > > rough > > > > > > > > > migration > > > > > > > > > path > > > > > > > > > > > > > > > > > > --- > > > > > > > > > > > > > > > > > > Migrating from JUnit4 to JUnit5 can be done in some steps, > > so that > > > > we > > > > > > > > > can > > > > > > > > > gradually move from JUnit4 to JUnit5. > > > > > > > > > > > > > > > > > > 0. (There is a way to use JUnit4 + 5 at the same time in a > > project - > > > > > > > > > you'd > > > > > > > > > use a specific JUnit4 runner to execute JUnit5. I'd like to > > skip > > > > this > > > > > > > > > step > > > > > > > > > as it would slow down migration significantly) > > > > > > > > > 1. Use JUnit5 with vintage runner. JUnit4 tests run mostly > > out of > > > > the > > > > > > > > > box. > > > > > > > > > The most important difference is that only 3 base rules are > > > > supported > > > > > > > > > and > > > > > > > > > the remainder needs to be migrated. Luckily, most of our > > rules > > > > derive > > > > > > > > > from > > > > > > > > > the supported ExternalResource. So in this step, we would > > need to > > > > > > > > > migrate > > > > > > > > > the rules. > > > > > > > > > 2. Implement new tests in JUnit5. > > > > > > > > > 3. Soft-migrate old tests in JUnit5. This is mostly a > > renaming of > > > > > > > > > annotation (@Before -> @BeforeEach, etc.). Adjust > > parameterized > > > > tests > > > > > > > > > (~400), replace rule usages (~670) with extensions, exception > > > > handling > > > > > > > > > (~1600 tests), and timeouts (~200). This can be done on a > > test class > > > > > by > > > > > > > > > test class base and there is no hurry. > > > > > > > > > 4. Remove vintage runner, once most tests are migrated by > > doing a > > > > > final > > > > > > > > > push for lesser used modules. > > > > > > > > > > > > > > > > > > Let me know what you think and I'm happy to answer all > > questions. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > Arvid Heise | Senior Java Developer > > > > > > > > <https://www.ververica.com/> > > > > > > > > Follow us @VervericaData > > > > > > > > -- > > > > > > > > Join Flink Forward <https://flink-forward.org/> - The Apache Flink > > > > Conference > > > > > > > > Stream Processing | Event Driven | Real Time > > > > > > > > -- > > > > > > > > Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany > > > > > > > > -- > > > > Ververica GmbH > > > > Registered at Amtsgericht Charlottenburg: HRB 158244 B > > > > Managing Directors: Timothy Alexander Steinert, Yip Park Tung Jason, Ji > > > > (Toni) Cheng > > > > > >
