We currently add @Ignore to failing unit tests so they don't cause our
builds to fail.

At some point in the future, we have to remember that a unit test was
written for a bug and remove the @Ignore.

Instead, we could write the unit test to pass with the current broken
behavior, and fail if the correct behavior is observed. The fail message
could be "If this test passes, it means Bug 12345 is resolved. Please
change this unit test to pass for correct behavior and close the bug."

Does this seem like a good idea? Is there a way we can avoid wrapping
failing unit tests in try/catch blocks to handle this, instead using some
annotation like @OpenBug(bug=12345, openIfThrows=[IllegalStateException]),
which fails with above message if an Exception class in openIfThrows is not
thrown. Converting this unit test to work for broken behavior to correct
behavior only requires removing the annotation.

There are a couple OpenBugzillaIssues.java unit test suites, which probably
allow non-Ignored tests to fail without breaking the build, but doesn't
alert us when a bug passes without reviewing the logs.

Reply via email to