[
https://issues.apache.org/jira/browse/STDCXX-683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12553961
]
Andrew Black commented on STDCXX-683:
-------------------------------------
I think that a distinction between expected and unexpected NOUT states should
be highlighted. This would permit us to easily spot non-regression tests which
failed to produce any output. Without an XNOUT state, it would be necessary to
check if each test is a regression test before looking past an NOUT row. In
some cases, the regression notation (.stdcxx-NNN) may be trimmed, due to the
length of the test name, making it harder to determine if a test is a
regression test. Perhaps it would make more sense to use XNOUT for unexpected
NOUT results and give it a yellow or red background.
An unrelated thought I'm having is that it might make sense to check how
accessible the color scheme is to someone who is red/green color blind. One
tool I tracked down that can be used is http://colorfilter.wickline.org/ . The
observation I'm having is that the dark green background used for the OK/XPASS
states appears the same as the background used for the [SIG]<name> state.
> implement notion of expected failures in the test suite
> -------------------------------------------------------
>
> Key: STDCXX-683
> URL: https://issues.apache.org/jira/browse/STDCXX-683
> Project: C++ Standard Library
> Issue Type: New Feature
> Components: Test Driver, Tests
> Affects Versions: 4.2.0
> Reporter: Martin Sebor
> Assignee: Martin Sebor
> Priority: Critical
> Attachments: codes.html, xcodes.html
>
>
> Tests (or examples) that fail for known reasons that we haven't been able to
> deal with should be distinguished from failures that haven't been analyzed
> yet. For example, an example program that fails to compile on an older target
> platform because of a compiler bug that we can't find a simple/elegant
> workaround should be flagged as such in the test results. Similarly, a test
> that fails one or more assertions due to compiler or libc bugs on a specific
> platform (or a set of platforms) that we are unable to work around should be
> reported as such.
> This is important in order to reduce the currently fairly large number of
> unexpected failures and to be able to make changes without having to worry
> about regressions as much.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
