Good idea. I have created https://github.com/apache/pulsar/pull/9398 for adding the issue template for reporting flaky tests.
BR, Lari On Sat, Jan 30, 2021 at 8:51 PM Sijie Guo <guosi...@gmail.com> wrote: > > I've been > using this type of template for reporting a flaky test: > https://gist.github.com/lhotari/a5c67359b362b4f3d8729330d65a2298 . > > > Can you add an issue template to Pulsar? > > - Sijie > > On Fri, Jan 29, 2021 at 10:27 AM Lari Hotari <lari.hot...@sagire.fi> > wrote: > > > Dear Pulsar community members, > > > > In order to improve our CI, we will have to fix the flaky tests. In some > > cases it might be necessary to replace an existing test with a redesigned > > test. > > > > The draft PIP "Changes to flaky test handling" document > > < > > > https://docs.google.com/document/d/10lmn4pW1IsT_8D1ZE0vMjASX0HhjdGdjB794iyScwns/edit?usp=sharing > > > > > lists > > the top 10 flaky tests. A lot of them have already been address by pull > > requests in the past week or so. > > > > This is the list of recent PRs that fix flaky tests from the top 10 flaky > > tests list: > > https://github.com/apache/pulsar/pull/9286 > > https://github.com/apache/pulsar/pull/9243 > > https://github.com/apache/pulsar/pull/9258 > > https://github.com/apache/pulsar/pull/9356 > > > > These are the GH issues for the remaining ones in the top 10 flaky tests > > list: > > https://github.com/apache/pulsar/issues/6368 > > https://github.com/apache/pulsar/issues/9369 > > https://github.com/apache/pulsar/issues/9368 > > > > If you would like to help to fix flaky tests you can pick one of the open > > issues above. Just add a comment on the issue when you start working on > it > > so that we can coordinate activities. > > > > It is also helpful to report a flaky test when you encounter one. I've > been > > using this type of template for reporting a flaky test: > > https://gist.github.com/lhotari/a5c67359b362b4f3d8729330d65a2298 . The > > issues #9368 and #9369 have been reported using this template. > > Search for the test name before reporting so that we don't end up with > > duplicates. > > > > The issues #6368, #9369 and #9368 are the 3 next important issues to fix. > > I'm planning to create a more extensive list of the flaky failures so > that > > we can target the most flaky ones when we continue fixing the flaky > tests. > > I have some scripts in development to assist in mining the Pulsar Github > > Action workflow run logs. > > > > This is a search to find flaky issues in Pulsar GH issues: > > > > > https://github.com/apache/pulsar/issues?q=flaky+sort%3Aupdated-desc+is%3Aopen > > > > Looking forward to the contributions for fixing flaky tests, > > > > BR, > > > > Lari > > >
