Hi guys, As a result of a situation formed around dnssec tests and one of the long-term bugs in dnssec feature [1], I'd like to share some general considerations with the whole team. First I'll state a couple of obvious things just to eliminate all possible fundamental disagreements. 1. The biggest purpose of test automation is to quickly find regressions in existing, released features 2. In order for the automated tests to effectively catch these regressions, the testsuite must 100% pas, so that any introduced issue will be noticed upon the very first test failure
3. Now, the standard workflow of using of automated tests is as follows: What happens, when a regression is found? Right, it's being quickly fixed, a new release is published, the test is green again, everybody is happy. What happens, if a team does not have enough resources to quickly (like within a week or so) fix this bug? Well. that's also quite a common situation. The company then elaborates a workaround, issues a notice with description of this bug and a workaround so that customers know about it and it would not affect their workflows. The tests are then tweaked accordingly with this known workaround automated so that each run is green again and ready to catch the next regression. Now, what happens if the tests are NOT tweaked? The team then has constantly "red" testruns which are not able to catch a next possible regression. And it inevitably occurs. Being uncaught by a team's CI it creeps into production and reach customers, who (quite expectedly) become nervous, because they would expect to receive at least a notice from the vendor, that the feature they are using has some more bugs in it. Guys, I am really sorry for being Captain Obvious here, but it seems that most of our team puts the principle "No workarounds, we need to fix the bug no matter what" above any common sense. [1] https://fedorahosted.org/freeipa/ticket/5348 -- Oleg Fayans Quality Engineer FreeIPA team RedHat. -- Manage your subscription for the Freeipa-devel mailing list: https://www.redhat.com/mailman/listinfo/freeipa-devel Contribute to FreeIPA: http://www.freeipa.org/page/Contribute/Code
