Done, r212762.
On Thu, Jul 10, 2014 at 1:02 PM, Todd Fiala <[email protected]> wrote: > Hey all, > > I had been operating under the assumption that a test marked XFAIL > (expected failure) that passes (which gets categorized as an "unexpected > success") would fail the test run. I have recently learned in the last few > days that this is not true - the unexpected success does get captured in > the logs and does show up as a 'u' in the test run, but it doesn't cause > the entire test process exit code to fail in the way that one or more > failed tests would. > > Given that, I'm going to flip the tests that were intermittent to be XFAIL > instead of skip. This allows the tests to still run, allowing the code to > run, and enabling us to catch (as an "Error") if the test actually seg > faults or something. All things being equal, I'd prefer to get notice this > way over hiding the unexpected success. (Note we have a different issue in > the test runner where we don't always get something useful when we > segfault, but I've already filed a bug on that). > > I'd be in favor of doing more to track the unexpected successes --- if we > have tests marked as XFAIL that have since been fixed and always succeed, > it would be good to flip them from XFAIL to normal test status that is > expected to pass. But we can address that later, maybe with buildbots that > can track those over time and start generating "potentially passing XFAILS" > reports if they pass all of the last x runs (with some largish x). > > I'll go ahead and flip the intermittent skipped tests that I changed > recently to XFAIL (generally on MacOSX and Linux) to adhere to this idea. > -- > -Todd > -- -Todd
_______________________________________________ lldb-dev mailing list [email protected] http://lists.cs.uiuc.edu/mailman/listinfo/lldb-dev
