On 1/10/12 Jan 10 -5:51 PM, Jeff Cunningham wrote: > On 01/10/2012 04:00 AM, Anton Vodonosov wrote: >> Hello. >> >> For my common lisp testing project I aggregate results of a library >> test suite into single value - ok/fail. >> >> I just tested ECL and have the following output from CFFI test suite: >> >> 4 out of 228 total tests failed: DEFCFUN.NOOP, CALLBACKS.BFF.1, >> STRING.ENCODING.UTF-16.BASIC, >> STRING.ENCODINGS.ALL.BASIC. >> No unexpected failures. >> >> What meaning do you put into the term "expected failure"? Does it mean >> the library is buggy, but >> these bugs are known? Or it means that some non-required features are >> absent, but the >> library in general OK? >> >> I am interested in both short answer - as a library author, how do you >> think CFFI test suite >> should be marked if only expected failures present - OK or FAIL? >> >> And also I am curious in this concrete example, what these 4 failures >> mean for CFFI on ECL? >> > > In a testing scenario, "expected failure" to me means the test was > designed to fail and it did. Usually, these are set up to test error > handling. In a large testing environment we would periodically toss in a > couple tests we knew would fail - one thing it tests it that the people > running the tests are actually running them. If they didn't come back > with these failures we knew there was a breakdown in the process.
That is not how the term is used in the CFFI tests, or in most of the unit testing libraries. Those libraries offer facilities for a test to PASS if and only if the code raises an error (or an error of a particular type). We should not conflate the notion of "raise an error" (or, for that matter, "return NIL") with the notion of "fail a test." Tests are about verifying expectations, and we can have valid, test-passing expectations that code will raise an error. best, r _______________________________________________ cffi-devel mailing list cffi-devel@common-lisp.net http://lists.common-lisp.net/cgi-bin/mailman/listinfo/cffi-devel
