Re: [racket-users] unit testing output
David Storrs wrote on 11/8/18 11:10 AM: Will test successes show output as well? For me that's the hard stop on using RackUnit -- the fact that it shows nothing on success means that there's no way to tell the difference between "everything ran and passed", "some things ran and passed but some things that should have run did not", and "nothing ran". I've felt the same about this -- it still bothers me a little. (Examples: In DrRacket, there's always a possibility that test-running has been disabled in a preference setting that's buried in a nonintuitive place, several hops deep in UI navigation, not a top-level indicator, but that someone might turn off accidentally when trying to adjust a different behavior. Or, on the command line, you might not realize you're in the wrong directory to pick up the tests you're expecting. Or maybe you'd like at least a crude idea of how much tests are running for a bit of unfamiliar code.) Perhaps the difference between "test failures" and "code errors" here (neither of which you'll see in DrRacket, if there's not a problem) is that, when you run code as a program, there is usually some other behavior you can see, to know that at least some code has been processed. Not so for passing tests. My original portable Scheme test framework always printed a full log, including ending with a summary like: ;;; END "Foo Station" TESTS: FAILED ;;; (Total: 4 Passed: 3 Failed: 1) My later Racket-specific test framework, Overeasy, used the logger for non-test-failure messages. But probably the Racket logs are not normally visible to most programmers, so this isn't great. Absent a richer test metadata mechanism in DrRacket and `raco test`, I'm not sure how to do this better for both DrRacket and command line. So... Would the following simple two-part approach (special-casing DrRacket) work, or does someone have a better idea? * When tests are under current DrRacket, we could use stdout for a one-line successful tests summary message, and stderr for a one-line tests failure summary message of the same format. The different streams are so that we avoid misusing the HCI effect that DrRacket achieves when it colors errors in angry red. (I think this doesn't break much, since programs under DrRacket won't be using stdout and stderr for old Unix "software tools" composition anyway.) * When tests are run under `raco test`, we could use either stdout or stderr for the one-line summary message. The question then is whether any CI-like frameworks that run tests like `raco test` currently treating any output as failure (I'm not certain, offhand). We could always kludge an environment variable to inhibit the summary for unusual situations, but I'd like to keep this kind of thing simple and clean. An alternative to the above would be to expose test metadata from the `test` submodules somehow, such as with "push" calls out akin to how `error-display-handler` works, or by a convention of `test` submodules providing a "pull" interface for programs like DrRacket to query. Making that work requires changes to core Racket code. (And, as usual with this kind of facility, probably half of everyone will think it's being done wrong.) The simple stdout/stderr convention above, special-casing DrRacket, OTOH, can be done easily, in a distributed and non-disruptive way. -- You received this message because you are subscribed to the Google Groups "Racket Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to racket-users+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [racket-users] unit testing output
On Wed, Nov 7, 2018 at 3:45 PM Neil Van Dyke wrote: > Greg Hendershott wrote on 11/6/18 9:56 AM: > > I'd suggest writing to stderr the same location format used by rackunit. > > OK, thank you (and David Storr). I looked at the current RackUnit. How > about the following proposal? > > Proposal: A test failure message is a one-line message that: > * in DrRacket, includes the clickable error icons; and > * on the command line (or in Emacs, Vim, etc.), is to stderr, and > includes the filename+line+column in a conventional non-Racket-specific > format. > > Details... > > The reason I don't want to use the RackUnit message format exactly is > that it has some excess stuff in it that won't work great with reporting > many tests, like I tend do them in Overeasy. Consider, for example, a > dozen tests failing, and you can eyeball the dozen one-line messages at > once to see whether they appear all related. > > To implement the DrRacket clickable icons, it looks from > "racket-7.1/share/racket/pkgs/rackunit-lib/rackunit/private/format.rkt" > like I can do something like this simplified demo: > > #lang racket/base > (with-handlers ((exn:fail? (lambda (exn) > ((error-display-handler) (exn-message > exn) exn >(error "TEST-FAILURE: [foomodule barproc empty-string] Expected > 42, got 0.")) > > In DrRacket, that demo displays a concise single-line message, with the > clickable icons: > > TEST-FAILURE: [foomodule barproc empty-string] Expected 42, > got 0. > > From the command line, that demo displays multi-line, which I don't > want (mainly for the dozen-failures-eyeballing reason): > > TEST-FAILURE: [foomodule barproc empty-string] Expected 42, got 0. >context...: > "/home/user/paddle-test-play.rkt": [running body] > temp37_0 > for-loop > run-module-instance!125 > perform-require!78 > > So, how about the command line format omits the calling stack context, > and instead uses a simple conventional one-line format that works out of > the box with most tools, like one of: > > PROGRAM:FILE:LINE:COL: MESSAGE > PROGRAM:FILE:LINE: MESSAGE > FILE:LINE:COL: MESSAGE > FILE:LINE: MESSAGE > > where "PROGRAM" (if present) *might* be something like "TEST" or > "TEST-FAILURE", instead of "my-racket-test-engine" or "raco test" or > whatever Racket program invoked the tests. > > BTW, that "[foomodule barproc empty-string]" part of the failure message > is Overeasy's `test-section` symbolic way of identifying tests, which is > complementary to source location, and meaningful to programmers/testers: > https://www.neilvandyke.org/racket/overeasy/#%28part._.Test_.Sections%29 > > BTW, the "Expected ___, got ___." message would also have other forms, > for different test pass criteria or ways that tests could fail. That > simple form is just for the most common case, of two sets of normal > expression result values failing a equivalence check by `equal?`. > > Will test successes show output as well? For me that's the hard stop on using RackUnit -- the fact that it shows nothing on success means that there's no way to tell the difference between "everything ran and passed", "some things ran and passed but some things that should have run did not", and "nothing ran". -- > You received this message because you are subscribed to the Google Groups > "Racket Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to racket-users+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "Racket Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to racket-users+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [racket-users] unit testing output
Greg Hendershott wrote on 11/6/18 9:56 AM: I'd suggest writing to stderr the same location format used by rackunit. OK, thank you (and David Storr). I looked at the current RackUnit. How about the following proposal? Proposal: A test failure message is a one-line message that: * in DrRacket, includes the clickable error icons; and * on the command line (or in Emacs, Vim, etc.), is to stderr, and includes the filename+line+column in a conventional non-Racket-specific format. Details... The reason I don't want to use the RackUnit message format exactly is that it has some excess stuff in it that won't work great with reporting many tests, like I tend do them in Overeasy. Consider, for example, a dozen tests failing, and you can eyeball the dozen one-line messages at once to see whether they appear all related. To implement the DrRacket clickable icons, it looks from "racket-7.1/share/racket/pkgs/rackunit-lib/rackunit/private/format.rkt" like I can do something like this simplified demo: #lang racket/base (with-handlers ((exn:fail? (lambda (exn) ((error-display-handler) (exn-message exn) exn (error "TEST-FAILURE: [foomodule barproc empty-string] Expected 42, got 0.")) In DrRacket, that demo displays a concise single-line message, with the clickable icons: TEST-FAILURE: [foomodule barproc empty-string] Expected 42, got 0. From the command line, that demo displays multi-line, which I don't want (mainly for the dozen-failures-eyeballing reason): TEST-FAILURE: [foomodule barproc empty-string] Expected 42, got 0. context...: "/home/user/paddle-test-play.rkt": [running body] temp37_0 for-loop run-module-instance!125 perform-require!78 So, how about the command line format omits the calling stack context, and instead uses a simple conventional one-line format that works out of the box with most tools, like one of: PROGRAM:FILE:LINE:COL: MESSAGE PROGRAM:FILE:LINE: MESSAGE FILE:LINE:COL: MESSAGE FILE:LINE: MESSAGE where "PROGRAM" (if present) *might* be something like "TEST" or "TEST-FAILURE", instead of "my-racket-test-engine" or "raco test" or whatever Racket program invoked the tests. BTW, that "[foomodule barproc empty-string]" part of the failure message is Overeasy's `test-section` symbolic way of identifying tests, which is complementary to source location, and meaningful to programmers/testers: https://www.neilvandyke.org/racket/overeasy/#%28part._.Test_.Sections%29 BTW, the "Expected ___, got ___." message would also have other forms, for different test pass criteria or ways that tests could fail. That simple form is just for the most common case, of two sets of normal expression result values failing a equivalence check by `equal?`. -- You received this message because you are subscribed to the Google Groups "Racket Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to racket-users+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [racket-users] unit testing output
On Fri, Nov 2, 2018 at 3:38 PM Neil Van Dyke wrote:
> * Should such a failing message also have some way of click navigating
> to the test failure in DrRacket, like for exceptions? (Note that the
> answer to the next might be involved.)
Yes please! It is no fun finding a failing test source in an
unfamiliar code base.
I'd suggest writing to stderr the same location format used by rackunit.
That way people in emacs can add to compilation-error-regexp-alist items like:
;; rackunit check-xxx
("#]+\\)> \\([0-9]+\\) \\([0-9]+\\)" 1 2 3)
;;rackunit/text-ui test-suite
("^location:[ ]+\\(\\([^:]+\\):\\([0-9]+\\):\\([0-9]+\\)\\)" 2 3 4 2 1)
And people using other editors can do the equivalent.
--
You received this message because you are subscribed to the Google Groups
"Racket Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to racket-users+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Re: [racket-users] unit testing output
On Fri, Nov 2, 2018 at 3:38 PM Neil Van Dyke wrote:
> Anyone have info or preferences on what are good output conventions for
> Racket package embedded unit tests? (For use with all of: `raco test`,
> package catalog and other builds, DrRacket and other IDEs?)
>
> I intend to reimplement Overeasy
> ("https://www.neilvandyke.org/racket/overeasy/;) in a way that is even
> more lightweight in both ease-of-use and in implementation. The syntax
> and features will be very similar, for now (there's one innovative big
> feature I might add later), but what I'm not yet sure about is all the
> details of the output.
>
> For output, here's what I'm currently thinking:
>
> * Most test messages other than fails are sent to logger `debug@test`.
>
> * A test failing unexpectedly (or succeeding unexpectedly) results in a
> message to stderr.
>
Maybe instead use "info@test-pass" and "info@test-fail" ? These messages
are expected output, not debug output, so should probably not print at the
debug level. Also, putting them on separate loggers would offer
finer-grained control.
> * Should such a failing message also have some way of click navigating
> to the test failure in DrRacket, like for exceptions? (Note that the
> answer to the next might be involved.)
>
> * Should the first failing test raise `exn:fail?`? Or should
> `exn:fail?` be raised at the end of all a module's tests if any failed?
> Or should `exn:fail?` not be raised at all by the tests?
>
The way I've handled it in test-more is that:
- Failing tests emit a message, not an exception
- Tests that throw an exception will terminate the script unless wrapped
in a test-suite. In that case the test-suite converts the exception to a
test failure and the rest of the tests in the suite are not run but
subsequent tests / test suites will be.
> * Should a test *failing expectedly* (as marked with `:fail`) also
> result in a message to stderr? Maybe a `warning@test` logger is
> better? (Note that, IIRC, the package catalog will regard output as a
> failure of tests for the package. Which can remind us that there is
> `:fail`, but can also discourage the programmer/qa capturing and
> tracking the info about the expected-fail tests.)
>
How difficult would it be to make the package server determine success /
failure by the return value of the test script as opposed to the
presence/absence of output? In my opinion output is pretty orthogonal to
success or failure, and I can think of times when it's desirable to have a
package emit output on load. (e.g. when it's a deprecated version that is
about to be sunset)
> * For the test-failure messages to stderr, what textual format should
> they have? More like the current Racket exception format, or current
> RackUnit, or Overeasy? (Or is there a pressing need for a different
> format?)
>
I'm obviously a fan of test-more's conventions:
#lang racket
(require handy/test-more)
(ok #f "(ok #f) fails and returns #f")
(ok 7 "(ok 7) succeeds and returns 7")
(is 7 8 "(is 7 8) prints the failure message and returns 7")
(lives (thunk 'ok) "(lives (thunk 'ok)) succeeds and returns 'ok")
(test-suite
"test-suite example"
(ok 'yes "passes")
(ok #f "fails")
(diag "Note that test-suite returns void, so the rests of tests inside a
test-suite do not show up on the output. Consider this an incentive to use
test-suites")
(raise 'error)
)
output is:
NOT ok 1 - (ok #f) fails and returns #f
#f
ok 2 - (ok 7) succeeds and returns 7
7
NOT ok 3 - (is 7 8) prints the failure message and returns 7
Got: 7
Expected: 8
7
ok 4 - (lives (thunk 'ok)) succeeds and returns 'ok
'ok
(START test-suite: test-suite example)
ok 5 - passes
NOT ok 6 - fails
### (Note that test-suite returns void, so the rests of tests inside a
test-suite do not show
up on the output. Consider this an incentive to use test-suites.)
NOT ok 7 - Exception thrown! Test message: 'test-suite completed without
throwing uncaught
exception'. Exception: 'error'
Got: #f
Expected: #t
'error
Total tests passed so far: 3
Total tests failed so far: 4
(END test-suite: test-suite example)
WARNING: Neither (expect-n-tests N) nor (done-testing) was called. May not
have run all
tests.
Having the tests return their values is a deliberate choice, as it's useful
for doing conditionals and embedded testing. Having test-suite eat those
values so that they don't show up on the output is for aesthetics.
The warning at the end can be suppressed either by using (expect-n-tests
7) or (done-testing). [NB: 'test-suite' is a test, which is why this is 7
tests instead of 6]
--
You received this message because you are subscribed to the Google Groups
"Racket Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to racket-users+unsubscr...@googlegroups.com.
For more options, visit
[racket-users] unit testing output
Anyone have info or preferences on what are good output conventions for
Racket package embedded unit tests? (For use with all of: `raco test`,
package catalog and other builds, DrRacket and other IDEs?)
I intend to reimplement Overeasy
("https://www.neilvandyke.org/racket/overeasy/;) in a way that is even
more lightweight in both ease-of-use and in implementation. The syntax
and features will be very similar, for now (there's one innovative big
feature I might add later), but what I'm not yet sure about is all the
details of the output.
For output, here's what I'm currently thinking:
* Most test messages other than fails are sent to logger `debug@test`.
* A test failing unexpectedly (or succeeding unexpectedly) results in a
message to stderr.
* Should such a failing message also have some way of click navigating
to the test failure in DrRacket, like for exceptions? (Note that the
answer to the next might be involved.)
* Should the first failing test raise `exn:fail?`? Or should
`exn:fail?` be raised at the end of all a module's tests if any failed?
Or should `exn:fail?` not be raised at all by the tests?
* Should a test *failing expectedly* (as marked with `:fail`) also
result in a message to stderr? Maybe a `warning@test` logger is
better? (Note that, IIRC, the package catalog will regard output as a
failure of tests for the package. Which can remind us that there is
`:fail`, but can also discourage the programmer/qa capturing and
tracking the info about the expected-fail tests.)
* For the test-failure messages to stderr, what textual format should
they have? More like the current Racket exception format, or current
RackUnit, or Overeasy? (Or is there a pressing need for a different
format?)
--
You received this message because you are subscribed to the Google Groups "Racket
Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to racket-users+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
