30 марта 2012 г. 22:16 пользователь Christopher Jones <christopher.jo...@oracle.com> написал: > > > On 3/29/12 3:00 PM, Alexey Shein wrote: >> >> Hi, internals! >> >> I've got a suggestion about refactoring our tests suite. I'd like to >> remove XFAIL institution and mark all failing tests just as FAIL. >> XFAIL has a problem that it hides attention from failing tests >> depending on not yet fixed bugs (most important), not yet implemented >> features (less important). >> Failed tests should make pain. They should bug you every day until you >> go and fix them. >> XFAILs serve now as a pain-killers, we've got about 50 of them in the >> repo, so devs (I assume) think this way: "It's failing, but it's >> EXPECTED to fail, so let's leave it as is". >> That's wrong thinking. Either tests are correct and if they fail you >> should fix the code and leave them failed until the code is fixed, or, >> if the tests are incorrect - fix the tests or remove them completely. > > > The XFAIL mechanism reflects the reality of open source that not all > bugs are fixed. We need a simple, low maintenance way to have a > 'clean' testsuite shipped which exhibits minimal noise so that users > don't waste time investigating known failures.
I'm trying to solve 2 different problems here: 1) Separate clean testsuite (new failed bugs) from known failed bugs (as you said) - XFAIL solves that 2) Keep devs' attention on known failures - XFAIL doesn't solve that. You remember about them when you run tests and if you want make attention at them. What I propose is a single *daily* newsletter saying "Hey, guys! We still have XFAIL bugs on 5.3 <list bugs>, 5.4 <list bugs> and master <list bugs>. Bye!" That will make some pressure, especially if those bugs have maintainers. > We do get constant notification of bugs assigned to us. I don't believe it has any impact on the fix rate. Hmm, that's different. You get a notification if there's some change on that bug (new comment/state changed/patch etc.). If bug didn't change for years, you won't get any notifications -> it's more likely you forget about it. > XFAIL also allows end users to see when something has broken that used > to work. Maybe, but not the best way, since it involves manual editing phpt source FAIL->XFAIL. Jenkins build failure notification solves it better. > If the system is being overused, feel free to call people out on it. > I don't think it should be used for unimplemented features long term. > XFAIL is a simple mechanism. Anything different like moving tests to > a special 'failed' directory adds burden. I don't belive we have > extra cycles for this, but would be happy to be proved wrong. Agree, that's a lot of work, need to try something else. The problem is here "what bugs need to be solved for release to be made?". We need to separate these somehow. XFAIL doesn't really helps here since it's just "bugs that are hard to solve" and it doesn't enforce any priority here. For 5.4 release Stas used wiki for keeping track of bugs stopping the release. -- Regards, Shein Alexey -- PHP Internals - PHP Runtime Development Mailing List To unsubscribe, visit: http://www.php.net/unsub.php
