Hi! > I've got a suggestion about refactoring our tests suite. I'd like to > remove XFAIL institution and mark all failing tests just as FAIL. > XFAIL has a problem that it hides attention from failing tests > depending on not yet fixed bugs (most important), not yet implemented > features (less important). > Failed tests should make pain. They should bug you every day until you > go and fix them.
Please note that we were in that position and we moved from there. So to move back, we need some argument about what's different this time from the place we were a year ago. > XFAILs serve now as a pain-killers, we've got about 50 of them in the > repo, so devs (I assume) think this way: "It's failing, but it's > EXPECTED to fail, so let's leave it as is". Leaving it as is was happening anyway. It's not like we had crowds of devs descending on test fails but ignoring xfails. Most of xfails were fails and were sitting there ignored for years. So the difference was "running constantly with 50 fails" or "having some xfails but detecting new fails easily since we have no or very little fails normally". The problem is exactly that there are no devs thinking like you imagine them to think. > from a file) and use that. Failing tests should not be hidden. They are not hidden. But they were not being fixed when they were just fails - only thing that happened is that we constantly run with tons of fails, so it was impossible to distinguish situation of "everything is fine" from "the build is FUBAR". > functions or not. We could also introduce "Incomplete" state like it's > done in PHPUnit for these tests. So what's the difference between xfail and incomplete? -- Stanislav Malyshev, Software Architect SugarCRM: http://www.sugarcrm.com/ (408)454-6900 ext. 227 -- PHP Internals - PHP Runtime Development Mailing List To unsubscribe, visit: http://www.php.net/unsub.php
