Hi, Yes, sounds useful, previously we used Test*UnfixedBugs classes for things that are not fixed yet. These are excluded in the CI runs, but we still only find out about fixed items when manually running this test-class. I just found one today which passed and indicated that an open bug was fixed in the meantime.
Junit has some support for expected exceptions, but it is fairly poor in features, but it may be sufficient for this use case. Adding a test-suite which looks at bugzilla will cause additional web-accesses during CI runs and might cause unstability if bugzilla is down/unreachable/flaky or changes web-address (just happened a year ago!)... I saw this being used locally at my company but am not sure if it is a good idea for an open source project where code is built all over the place. Dominik. On Sat, Jan 2, 2016 at 11:19 PM, Javen O'Neal <[email protected]> wrote: > We currently add @Ignore to failing unit tests so they don't cause our > builds to fail. > > At some point in the future, we have to remember that a unit test was > written for a bug and remove the @Ignore. > > Instead, we could write the unit test to pass with the current broken > behavior, and fail if the correct behavior is observed. The fail message > could be "If this test passes, it means Bug 12345 is resolved. Please > change this unit test to pass for correct behavior and close the bug." > > Does this seem like a good idea? Is there a way we can avoid wrapping > failing unit tests in try/catch blocks to handle this, instead using some > annotation like @OpenBug(bug=12345, openIfThrows=[IllegalStateException]), > which fails with above message if an Exception class in openIfThrows is not > thrown. Converting this unit test to work for broken behavior to correct > behavior only requires removing the annotation. > > There are a couple OpenBugzillaIssues.java unit test suites, which probably > allow non-Ignored tests to fail without breaking the build, but doesn't > alert us when a bug passes without reviewing the logs. >
