30 марта 2012 г. 3:19 пользователь Stas Malyshev <smalys...@sugarcrm.com> написал: > Hi! > >> I've got a suggestion about refactoring our tests suite. I'd like to >> remove XFAIL institution and mark all failing tests just as FAIL. >> XFAIL has a problem that it hides attention from failing tests >> depending on not yet fixed bugs (most important), not yet implemented >> features (less important). >> Failed tests should make pain. They should bug you every day until you >> go and fix them. > > Please note that we were in that position and we moved from there. So to > move back, we need some argument about what's different this time from > the place we were a year ago. >> XFAILs serve now as a pain-killers, we've got about 50 of them in the >> repo, so devs (I assume) think this way: "It's failing, but it's >> EXPECTED to fail, so let's leave it as is". > > Leaving it as is was happening anyway. It's not like we had crowds of > devs descending on test fails but ignoring xfails. Most of xfails were > fails and were sitting there ignored for years. So the difference was > "running constantly with 50 fails" or "having some xfails but detecting > new fails easily since we have no or very little fails normally". > The problem is exactly that there are no devs thinking like you imagine > them to think. >
The difference started from 5.3.9 release when we start to pay *much more* attention to tests. You now can cleanly see failing tests, it's not that huge list, so it's big difference. If the problem is only with finding new fails, we can use jenkins for that - it already detects new fails in builds and can mail it here, so they won't get unnoticed. It could also bug a commiter who failed the build - nice feature to have. The main idea I'm trying to say is that it's comfortable to live with XFAILs. That's why they live by years. They don't get make any pressure, we don't have a release rule "No failing tests", so they go from release to release until some hero will come up and fix them. By turning them into FAILs they become common problem, because they start to annoy everyone, so it's easier to collaborate on their fixing. >> from a file) and use that. Failing tests should not be hidden. > > They are not hidden. But they were not being fixed when they were just > fails - only thing that happened is that we constantly run with tons of > fails, so it was impossible to distinguish situation of "everything is > fine" from "the build is FUBAR". They are not hidden, but they don't *really* bother everyone. My (quite limited setup though) on master branch reports this: Number of tests : 12100 8194 Tests skipped : 3906 ( 32.3%) -------- Tests warned : 0 ( 0.0%) ( 0.0%) Tests failed : 3 ( 0.0%) ( 0.0%) Expected fail : 35 ( 0.3%) ( 0.4%) Tests passed : 8156 ( 67.4%) ( 99.5%) We have 3 failing tests and 35 xfails, I don't see any tons of fails here. Sorry, if I sound like a broken record, but if we need to fix those, we need to make more noise about that. >> functions or not. We could also introduce "Incomplete" state like it's >> done in PHPUnit for these tests. > > So what's the difference between xfail and incomplete? XFAIL - expected to fail test. If it's fails - then it's ok. That's how I understand it. Failing test should not be ok, it's an error. If you get used to not paying attention on failing tests, you're in dangerous situation. It's like a fairy tale about boy that cried "Wolf!". In the end of the story nobody trusts him. That's why I think we should return trust back to failing tests. About incomplete, well, it seems it doesn't suite here much, it's about that test is not fully written or finished. For example, if we have a plan for some release branch (say, 5.4) about implementing some features, we can have failing/incomplete acceptance tests for those (in a separate suite for example), so release is just a matter of making all tests pass. If feature is quite big and can take several releases (traits come to my mind) it could always be put into separate branch until it's ready. -- Regards, Shein Alexey -- PHP Internals - PHP Runtime Development Mailing List To unsubscribe, visit: http://www.php.net/unsub.php
