> > > Please note that we were in that position and we moved from there. So to > move back, we need some argument about what's different this time from > the place we were a year ago. > > could you elaborate on this part? where were we a year ago?
> > XFAILs serve now as a pain-killers, we've got about 50 of them in the > > repo, so devs (I assume) think this way: "It's failing, but it's > > EXPECTED to fail, so let's leave it as is". > > Leaving it as is was happening anyway. It's not like we had crowds of > devs descending on test fails but ignoring xfails. Most of xfails were > fails and were sitting there ignored for years. So the difference was > "running constantly with 50 fails" or "having some xfails but detecting > new fails easily since we have no or very little fails normally". > The problem is exactly that there are no devs thinking like you imagine > them to think. > > yeah, but as we did see, the current approach makes it very easy to "hide" even the not so small issues (for example the bunch of date related XFAILS which you personally asked multiple times to be fixed before the 5.4 release). I think that in it's current form XFAIL hurts more than it helps. > > from a file) and use that. Failing tests should not be hidden. > > They are not hidden. But they were not being fixed when they were just > fails - only thing that happened is that we constantly run with tons of > fails, so it was impossible to distinguish situation of "everything is > fine" from "the build is FUBAR". > yeah, and the exact same thing happened with the 5.3.7/CVE-2011-3189, even though that we have XFAILs. I think that eliminating the failing tests and making the fails noisy would be a better approach. I think that those spontaneous "TestFests" initiated by Rasmus really helped, and now that we are on git/github, there would be even greater audience. > > > functions or not. We could also introduce "Incomplete" state like it's > > done in PHPUnit for these tests. > > So what's the difference between xfail and incomplete? > http://www.phpunit.de/manual/3.2/en/incomplete-and-skipped-tests.html currently the XFAIL can mean that either the test is incomplete, or the functionality is missing/the code is broken, but we are expecting that for some reason. killing the XFAIL and adding the Incomplete test feature would allow the first, but the second should be a failing test. -- Ferenc Kovács @Tyr43l - http://tyrael.hu
