Deleting ignored tests does lead us to losing the reason as to why the test case was around so I would rather keep it around. I think it would be more valuable to generate a report that goes on the website/wiki showing stability of the modules (num tests, num passed, num skipped, num failed (running averages over the past N runs)). We had discussed doing something like this for ValidatesRunner so we could show which runner supports what automatically.
On Mon, May 11, 2020 at 12:53 AM Jan Lukavský <je...@seznam.cz> wrote: > I think that we do have Jira issues for ignored test, there should be no > problem with that. The questionable point is that when test gets Ignored, > people might consider the problem as "less painful" and postpone the > correct solution until ... forever. I'd just like to discuss if people see > this as an issue. If yes, should we do something about that, or if no, > maybe we can create a rule that test marked as Ignored for long time might > be deleted, because apparently is only a dead code. > On 5/6/20 6:30 PM, Kenneth Knowles wrote: > > Good point. > > The raw numbers are available in the test run output. See > https://builds.apache.org/view/A-D/view/Beam/view/PostCommit/job/beam_PreCommit_Java_Cron/2718/testReport/ > for > the "skipped" column. > And you get the same on console or Gradle Scan: > https://scans.gradle.com/s/ml3jv5xctkrmg/tests?collapse-all > This would be good to review periodically for obvious trouble spots. > > But I think you mean something more detailed. Some report with columns: > Test Suite, Test Method, Jira, Date Ignored, Most Recent Update > > I think we can get most of this from Jira, if we just make sure that each > ignored test has a Jira and they are all labeled in a consistent way. That > would be the quickest way to get some result, even though it is not > perfectly automated and audited. > > Kenn > > On Tue, May 5, 2020 at 2:41 PM Jan Lukavský <je...@seznam.cz> wrote: > >> Hi, >> >> it seems we are accumulating test cases (see discussion in [1]) that are >> marked as @Ignored (mostly due to flakiness), which is generally >> undesirable. Associated JIRAs seem to be open for a long time, and this >> might generally cause that we loose code coverage. Would anyone have >> idea on how to visualize these Ignored tests better? My first idea would >> be something similar to "Beam dependency check report", but that seems >> to be not the best example (which is completely different issue :)). >> >> Jan >> >> [1] https://github.com/apache/beam/pull/11614 >> >>
