Hi guys,

As a result of a situation formed around dnssec tests and one of the
long-term bugs in dnssec feature [1], I'd like to share some general
considerations with the whole team.
First I'll state a couple of obvious things just to eliminate all
possible fundamental disagreements.
1. The biggest purpose of test automation is to quickly find regressions
in existing, released features
2. In order for the automated tests to effectively catch these
regressions, the testsuite must 100% pas, so that any introduced issue
will be noticed upon the very first test failure

3. Now, the standard workflow of using of automated tests is as follows:

What happens, when a regression is found? Right, it's being quickly
fixed, a new release is published, the test is green again, everybody is
What happens, if a team does not have enough resources to quickly (like
within a week or so) fix this bug? Well. that's also quite a common
situation. The company then elaborates a workaround, issues a notice
with description of this bug and a workaround so that customers know
about it and it would not affect their workflows. The tests are then
tweaked accordingly with this known workaround automated so that each
run is green again and ready to catch the next regression.

Now, what happens if the tests are NOT tweaked? The team then has
constantly "red" testruns which are not able to catch a next possible
regression. And it inevitably occurs. Being uncaught by a team's CI it
creeps into production and reach customers, who (quite expectedly)
become nervous, because they would expect to receive at least a notice
from the vendor, that the feature they are using has some more bugs in it.

Guys, I am really sorry for being Captain Obvious here, but it seems
that most of our team puts the principle "No workarounds, we need to fix
the bug no matter what" above any common sense.

[1] https://fedorahosted.org/freeipa/ticket/5348

Oleg Fayans
Quality Engineer
FreeIPA team

Manage your subscription for the Freeipa-devel mailing list:
Contribute to FreeIPA: http://www.freeipa.org/page/Contribute/Code

Reply via email to