[ 
https://issues.apache.org/jira/browse/STDCXX-683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12553815
 ] 

Andrew Black commented on STDCXX-683:
-------------------------------------

Looking at the revised xcodes.html file, I think it will be necessary to add an 
'XNOUT' (expected NOUT) and 'XFMAT'  (expected FMAT) assertion code.  This is 
because we currently have tests which currently produce NOUT and FMAT status 
messages, and these messages are expected.  Examples include NOUT messages from 
many regression tests (like 18.limits.stdcxx-436) and FORMAT (FMAT) messages 
from some driver tests (0.cmdopts, 0.strncmp, and  0.valcmp).

> implement notion of expected failures in the test suite
> -------------------------------------------------------
>
>                 Key: STDCXX-683
>                 URL: https://issues.apache.org/jira/browse/STDCXX-683
>             Project: C++ Standard Library
>          Issue Type: New Feature
>          Components: Test Driver, Tests
>    Affects Versions: 4.2.0
>            Reporter: Martin Sebor
>            Assignee: Martin Sebor
>            Priority: Critical
>         Attachments: codes.html, xcodes.html
>
>
> Tests (or examples) that fail for known reasons that we haven't been able to 
> deal with should be distinguished from failures that haven't been analyzed 
> yet. For example, an example program that fails to compile on an older target 
> platform because of a compiler bug that we can't find a simple/elegant 
> workaround should be flagged as such in the test results. Similarly, a test 
> that fails one or more assertions due to compiler or libc bugs on a specific 
> platform (or a set of platforms) that we are unable to work around should be 
> reported as such.
> This is important in order to reduce the currently fairly large number of 
> unexpected failures and to be able to make changes without having to worry 
> about regressions as much.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Reply via email to